Re: Testing a GUI application

Before I begin what will be a fairly long post, I wanted to take a moment and say thanks for posting this question. It's really made me closely examine and rethink some of my GUI testing practices. It also gave me the opportunity to experiment with Prima, which looks very impressive. Here goes nothing...

To borrow a phrase from Shrek and paraphrase BrowserUk, GUIs are like Onions - they have many layers. Testing can be done at each of these layers depending on your resources, level of committment, and expertise, but it can be time consuming and painful. Often, I find that I spend at least as much time (usually more) coding test cases as I do writing code, but usually I find the time was well spent.

Tools can help significantly, but often they have a price tag of their own in terms of money or time spent learning to use them. I've seen how a test tool in the hands of someone who knows how to use it can be invaluable and save a lot of time, unfortunately it's something that I don't have a lot of hands-on experience with. In the case of a large monolithic script, I agree with BrowserUk that a tool may be the best and perhaps most cost-effective way to go, but what if the script was not monolithic and was partitioned into Components which could each be tested as a unit?

I would argue that a careful design can greatly facilitate unit-testing, and at the very least serve to supplement testing through external record and playback. Do the design changes introduce a bit more complexity? It depends on the situation and on what you feel is complex. I'll try to explain the basic strategy that I've used in the past in the hope that you might adapt it to Prima if you think it's useful. If you're still interested, you may want to grab a cup of coffee or a Mountain Dew. This could take a while...

Layers

I've already mentioned layers, so I will define a few. They may or may not coincide with "standard" definitions, but at the very least they will serve as a useful lie for the purposes of this node.

Application/Business Logic - Usually the parts of the code that don't have too much to do with the GUI itself. This is the logic that the User Interface is supposed to make easier to use.

Toolkit Bindings - The code that binds your business logic to the toolkit library. Some individual I've talked to consider this to be merely the event handler code. For simplicity, I prefer to include in it all the code I write that uses code from the Toolkit library.

Toolkit Library/Window Manager Bindings - The code that interfaces with the Window Manager. This code can be very thin giving you direct access to the Window Manager, or it can be fairly thick and serve as an abstraction for one or more WindowManagers on different operating systems. Also, for simplicity, I include in this layer the language specific bindings (Perl, Ruby, Python, etc..) with the higher level toolkit library (if applicable).

Window Manager - Displays and manages graphical representations of programs and resources, and accepts events that signal user input or other changes to the windows environment.

Priorities

When I think of designing, coding, and testing a GUI application, my testing priorities start at the beginning of my list and work downward. I'm predominately interested in testing code that my team or I have written and not the code that it depends on. The main risk with this prioritization is that I could find a nasty bug somewhere in the Toolkit library or Window Manager layers, which may require non-trivial changes in the Toolkit Bindings layer which will already have test cases written. This can hurt when it is discovered late in process, but even in the event that this occurs, it's possible that it will not be too difficult to resolve. Depending on your situation or philosophy, your priorities may differ from mine.

If I have an already written application that doesn't lend itself to testing this way and refactoring isn't an option, then I would take the reverse approach. I would either find someone who can do the job well with some tool, or learn the tool myself. Ideally, testing can occur from both sides if my team has a resident tester who can document manual test case procedures and/or use a test tool.

Decompose the Application to Components

The strategy I use is simple: Break the application into smaller units that can be more readily tested. The lengths you go to partition your code will be a matter of preference and style. Some find that code split across multiple modules is difficult to maintain. I find very large files full of variables being defined all over the place difficult. Do what seems reasonable.

Separate the Application Logic

A good start, like you said in your post would be to:

...decouple the GUI from the program actions in order to produce a module that can be tested using one or many of the methods proposed on CPAN.

Lets say that I'm creating a very simple application that requires a user to Login. I might first write a simple script that captures the functionality I have in mind. For purpose of the example, it's not fancy.

use strict;
use Tk;

my use Tk;
use strict;

my $mw = MainWindow->new;
my $f = $mw->Frame->
   pack(-padx => 5, -pady => 5);

my $message = $f->Label->
   grid(-columnspan => 2, -sticky => 'ew');

my $userEntry = $f->Entry;
$f->Label(-text => "User:")->
   grid($userEntry, -sticky => 'e');
 
my $passwdEntry = $f->Entry(-show => '*');
$f->Label(-text => "Password:")->
   grid($passwdEntry, -sticky => 'e');

my $button = $f->Button(
   -text => "Login",
   -command => [\&login, $userEntry, $passwdEntry, $message]
)->grid(-columnspan => 2, -pady => 3);

MainLoop;

sub login
{
    my ($userE, $passE, $msg) = @_;

    if ($userE->get ne "eric" || $passE->get ne "stigisdead") {
       $userE->delete(0, 'end');
       $passE->delete(0, 'end');
       $msg->configure(-text => "Login Failed!");
    }  
    else {
       $msg->configure(-text => "Login Successful");
    }
}
[download]

It's important to remember that the code within the login would more likely be connecting to a database or an LDAP service and be using some secure mechanism to ensure the user's password isn't compromised. This is code that would be useful to test outside of the GUI, and could potentially be reused, so it makes sense to separate it in some way.

package AuthService;

sub login
{
   my ($user, $password) = @_;
   return 
      (defined($user) && defined($password) &&
      $user eq "eric" && $password eq "swordfish");
}
1;

package main;

use Test::More tests => 7;

ok(AuthService::login("eric", "swordfish"), 
   "good user/password combo");

ok(!AuthService::login("", "swordfish"), 
   "no user, good passwd");
ok(!AuthService::login(undef, "swordfish"), 
   "undef user, good passwd");

ok(!AuthService::login("eric", ""), 
   "good user, undef password");
ok(!AuthService::login("eric", undef), 
   "good user, no password");

ok(!AuthService::login("", ""), "Empty user/passord");
ok(!AuthService::login(), "undef user/password");
[download]

In the simple example, the overall impact so far isn't very positive: I've actually gained an additional line of code in my GUI script, and I have another Module to maintain! On the other hand, I can now test some of my code independant of the GUI, which is good. The benefit of this increases if AuthService contains a non-trivial implementation. Also, I'm in a good position if I decide I want to change the underlying implementation without having to dig through a lot of GUI code.

Repackage the Application as a Module

When looking for ways to directly access and test the GUI code, this step is crucial. The main idea here is that I want creation of the MainWindow and initiation of the event loop (MainLoop) in one place, and everything else in another, or perhaps several other locations. Does this mean that all the GUI code must be scattered across several files? No, not necessarily. You could still create a package for the app, at the very least it might require the GUI code to be split into two files, with one of those files having very little code. It could be as simple as: Create MainWindow, create application using MainWindow, start MainLoop (Prima maps to these basic steps, also.)

How should you package it? Should it be a large composite widget or some other custom class of your own devising? That choice is up to you. Either way will work fine. Whichever way you choose, the important thing will be to ensure that you have access to the GUI components that you will need for your tests. In Perl/Tk this is very easy using already available methods. Here's a first cut at what the repackaged code might look like. I've dropped the AuthService module for now to focus on the modified code.

## LoginForm.pm
package LoginForm;

use strict;
use base qw(Tk::Frame);
Tk::Widget->Construct('LoginForm');

sub Populate 
{
   my ($cw, $args) = @_;
   $cw->SUPER::Populate($args);

   my $f = $cw->Frame->pack;
   my $msg = $f->Label->
      grid(qw/-columnspan 2 -sticky ew/);

   my $userE = $f->Entry;
   $f->Label(-text => "User:")->
      grid($userE, -sticky => 'e');

   my $passE = $f->Entry(-show => '*');
   $f->Label(-text => "Password:")->
      grid($passE, -sticky => 'e');   

   my $button = $f->Button(
      -text => 'Login',
      -command => [Login => $cw]
   )->grid(qw/-columnspan 2 -pady 4/);

   $cw->Advertise(Message     => $msg);
   $cw->Advertise(UserEntry   => $userE);
   $cw->Advertise(PassEntry   => $passE);
   $cw->Advertise(LoginButton => $button);
} 

sub Login 
{
   my $cw = shift;
   my $user = $cw->Subwidget('UserEntry')->get;
   my $pass = $cw->Subwidget('PassEntry')->get;
   print "$user/$pass\n";
}
1;
  
## login.pl
use Tk;
use LoginForm;

my $mw = MainWindow->new;
$mw->LoginForm->pack;

MainLoop;
[download]

A full explanation of creating custom composites in Perl/Tk is beyond the scope of this discussion, and I've attempted to minimize the amount of code where possible. There are few things that bear a quick mention:

Tk::Widget->Construct('LoginForm'); this line dynamically creates a method in the Tk::Widget module called LoginForm, which can be used to create a new LoginForm instance. This is what permits the syntax used in main: $mw->LoginForm->pack;. I tend to be a little wary of its use it when I'm creating application-specific composites. If I were to omit this line, then I would need to use the following syntax to create my LoginForm: LoginForm->new($mw, %options);

my $f = $cw->Frame->pack; Observant Monks will have noticed that I'm extending Tk::Frame, yet one of the first things I do is create another Frame to house all my widgets. This is a subtle workaround to a minor issue that applies the color of the base Frame ($cw) to all of it's children after it is created. This can be irritating if you have set different default colors for the child widgets. Putting all the Components into another Frame protects their background/foreground colors from being changed.

$cw->Advertise(UserEntry => $userE); The Advertise method allows me to register the widget to an internal hash so that I can later access it through the Subwidget method. You can see it being used in Login.

Once you get over the initial shock of the modification, you'll notice that for the most part, the code is very close to the original script, and what has been gained? Quite a bit. I can now create an instance of this mini application, and because I have access to some of it's internals, I now have several options for running tests. In the case of Tk, Prima, and other toolkits as well, I don't necessarily even need to expose the subwidgets directly. Because I've created the MainWindow outside of the Module, I can use it to traverse the application's widget tree and locate widgets that way, as well. Another potential benefit is reuse. Depending on how I'm broken my application up into modules, there's a good chance that I may create something that can be used in another application, and save some work.

Watch for Dependancies

One potential barrier to reuse, and potentially testing is that once I reintegrate my AuthService module, then I've introduced a dependancy that might complicate testing of this GUI module. Dependancies are not always bad. It could be that you don't really intend to reuse the code, or if you do then you code where you're reusing it, may rely on the same dependancies. Regardless, depending on the toolkit you're using, you may have other options to reduce the coupling even further.

Your module may support configurable callbacks.
Your module may be dependant on an Abstract Base Class that defines all the methods, which can be implemented by a Mock instance of the class for test, or a real implementation that could also be tested independantly or integrated.

Here's how it might be decoupled in Tk:

package LoginForm;

use strict;
use base qw(Tk::Frame);

sub Populate 
{
   my ($cw, $args) = @_;
  $cw->SUPER::Populate($args);

   my $f = $cw->Frame->pack;
   my $msg = $f->Label->
      grid(qw/-columnspan 2 -sticky ew/);

   my $userE = $f->Entry;
   $f->Label(-text => "User:")->
      grid($userE, -sticky => 'e');

   my $passE = $f->Entry(-show => '*');
   $f->Label(-text => "Password:")->
      grid($passE, -sticky => 'e');   

   my $button = $f->Button(
      -text => 'Login',
      -command => [Login => $cw]
   )->grid(qw/-columnspan 2 -pady 4/);

   $cw->Advertise(Message     => $msg);
   $cw->Advertise(UserEntry   => $userE);
   $cw->Advertise(PassEntry   => $passE);
   $cw->Advertise(LoginButton => $button);

   $cw->ConfigSpecs(-logincmd => [qw/CALLBACK/]);
} 

sub Login 
{
   my $cw = shift;
   my $user = $cw->Subwidget('UserEntry')->get;
   my $pass = $cw->Subwidget('PassEntry')->get;
   my $msgL = $cw->Subwidget('Message'); 

   if ($cw->cget(-logincmd) && 
       $cw->Callback(-logincmd => $user, $pass)) 
   {
      $msgL->configure(-text => "Login Successful");
   }
   else
   {
      $msgL->configure(-text => "Login Failed!");
   }
}
1;
[download]

In the latest (and last) round of changes to LoginForm, I've created a new option that supports a Callback with the line: $cw->ConfigSpecs(-logincmd => [qw/CALLBACK/]);. I use this Callback within the Login method. If configured, it will be passed the username and password. Login expects the Callback to return a true or false value depending on whether or not the login was successful. This very simple step means that I can easily replace the login implementation easily, and makes it that much easier for me to test just the LoginForm.

Testing the Interface

Now, finally we move on to testing the LoginForm, but before I go too much further, there are some risks of taking this approach. By breaking things up into smaller pieces, there is a chance that all the individual pieces could test out fine, and then after I integrate the components, a few bugs rear their ugly head. It's also highly possible that I've missed a pathway that I didn't anticipate. Like BrowserUk wrote:

Users are not so easily tied down, and they are apt to see your GUI in completely different ways to you.

Given the vast number of combinations of keystrokes, mouseclicks, and other interactions, it's virtually impossible that you will be able to test them all. That's Ok. Be pragmatic about it and hit the cases that you can - something is usually better than nothing, and add others as time goes on. Don't test just the happy cases - try to trigger problems to test that they fail as expected.

Prior to starting testing, my advice would be document the cases that you are covering. It doesn't need to be very fancy, just a few words here and there - so that you, or others can easily review the cases that you have done. You might even consider putting these comments in POD and adding them within your test scripts. Just a thought...

Basic Testing - Call Event Handlers directly

These tests are fairly straightforward. You test by programmtically setting values within the various widgets, and then by calling any Event Handler subs directly, so see if the correct behavior occurs. This can be difficult if you've chosen to use nothing but anonymous subs for all of your event handling.

In Tk, your ability to test this way will also be hampered by the use of the XEvent object within your event handler code. Here's an example of two similar scripts that both bind to the same event. One uses Ev(@) and one uses XEvent within the event handler. Using the first is like having attributes extracted outside of the subrouting, and passed as a parameter. This means that we can simulate the call easily without really triggering an event. If you go the other route, then the only the event handler will work correctly will be if it receives an actual event.

Here is a sample test script to give a basic idea how testing is accomplished:

package AuthService;

sub login
{
   my ($user, $password) = @_;
   return 
      (defined($user) && defined($password) &&
      $user eq "eric" && $password eq "swordfish");
}
1;

use Test::More tests => 4;
use Tk;
use strict;
use LoginForm;

my $mw = MainWindow->new;

my $lf = LoginForm->new($mw, 
   -logincmd => \&AuthService::login
)->pack;
my $msg = $lf->Subwidget('Message');

#####################################
## No Parameters Set
#####################################
$lf->Login;
is($msg->cget(-text), "Login Failed!",
   "No user or password set");

#####################################
## Good credentials
#####################################
setUserPassword($lf, "eric", "swordfish");
$lf->Login;
is($msg->cget(-text), "Login Successful",
   "Valid credentials set");

#####################################
## No username set, good password
#####################################
setUserPassword($lf, "", "swordfish");
$lf->Login;
is($msg->cget(-text), "Login Failed!",
   "User not set, Password set");

#####################################
## Correct User set, no password
#####################################
setUserPassword($lf, "eric", "");
$lf->Login;
is($msg->cget(-text), "Login Failed!",
   "User set, no password");

$mw->destroy;

###############################################
## Convenience routines
###############################################

sub setUserPassword
{
   my ($lf, $user, $pass) = @_;
   clearLoginForm($lf);
   $lf->Subwidget('UserEntry')->insert(0, $user);
   $lf->Subwidget('PassEntry')->insert(0, $pass);
}

sub clearLoginForm
{
   my $lf = shift;
   $lf->Subwidget('Message')->configure(-text => "");
   $lf->Subwidget('UserEntry')->delete(0, 'end');
   $lf->Subwidget('PassEntry')->delete(0, 'end');
}
[download]

As you can see, setting up and running the test cases are fairly easy, but we really haven't done that good a job of testing the GUI. Some GUI components are involved, but so far, the tests are not receiving user events, yet.

Testing Using Toolkit-Generated Events

Most toolkits that I've used in the past have some way of generating events, and Tk and Prima are no different. In Tk, the usual way of programmatically dispatching user events is by calling the $widget->eventGenerate method.

The advantage of using the toolkit to fire events, is that your test script should be as portable as the toolkit you're using. That said, expect occasional annoyances with Tk, and be prepared to try different things when generating events. I find it useful to create higher level methods that hide the calls to eventGenerate so that I can create tests faster.

Here is a variation of the basic test script using eventGenerate:

package AuthService;

sub login
{
   my ($user, $password) = @_;
   return 
      (defined($user) && defined($password) &&
      $user eq "eric" && $password eq "swordfish");
}
1;
package main;

use strict;
use LoginForm;
use Test::More tests => 4;
use Tk;

my $mw = MainWindow->new;

my $lf = LoginForm->new($mw, 
   -logincmd => \&AuthService::login
)->pack;
my $msg = $lf->Subwidget('Message');

#####################################
## No Parameters Set
#####################################
$lf->Login;
is($msg->cget(-text), "Login Failed!",
   "No user or password set");

#####################################
## Good credentials
#####################################
simulateLogin($lf, "eric", "swordfish");
is($msg->cget(-text), "Login Successful",
   "Valid credentials set");

#####################################
## No username set, good password
#####################################
simulateLogin($lf, "", "swordfish");
is($msg->cget(-text), "Login Failed!",
   "User not set, Password set");

#####################################
## Correct User set, no password
#####################################
simulateLogin($lf, "eric", "");
is($msg->cget(-text), "Login Failed!",
   "User set, no password");

sub simulateLogin
{
   my ($lf, $user, $pass) = @_;
   clearLoginForm($lf);
   my $userE = $lf->Subwidget('UserEntry');
   my $passE = $lf->Subwidget('PassEntry');
   my $button = $lf->Subwidget('LoginButton');

   $lf->update;

   $userE->focusForce;
   setEntry($userE, $user);
   $userE->eventGenerate('<Tab>');
   setEntry($passE, $pass);
   $userE->eventGenerate('<Tab>');
   $button->eventGenerate('<Return>');
 }

sub clearLoginForm
{
   my $lf = shift;
   $lf->Subwidget('Message')->configure(-text => "");
   $lf->Subwidget('UserEntry')->delete(0, 'end');
   $lf->Subwidget('PassEntry')->delete(0, 'end');
}

sub setEntry 
{
   my ($entry, $text) = @_;
   foreach my $ch (split(//, $text)) {
      $entry->eventGenerate('<Key>', -keysym => $ch);
   }
}
[download]

A final tip when using this approach: it can sometimes help to inject a call to MainLoop into a procedure to get a visual snapshot and ensure that the current state reflects expectations.

Testing through the Window Manager

I rarely write test scripts at this level because I either run out of time, or it doesn't seem to have enough value to be worth the effort especially when i have the others layers of testing in place. Given my occasional problems with Tk's event generation, I'd probably consider it if I knew I was deploying to a particular platform and didn't have to worry about cross-platform concerns.

For this round, I'll try to test my Tk application with calls to Win32 Window Manager using Win32::GuiTest. One advantage that I have with both Tk and Prima, is that my widgets have access to their Win32 window handle (HWND), or their X Windows id. This can be retreived in Tk, through the $widget->id method, and Prima through the $widget->get_handle method.

package AuthService;

sub login
{
   my ($user, $password) = @_;
   return 
      (defined($user) && defined($password) &&
      $user eq "eric" && $password eq "swordfish");
}
1;
package main;

use strict;
use LoginForm;
use Test::More tests => 4;
use Tk;
use Win32::GuiTest qw(SendKeys SetForegroundWindow);

my $mw = MainWindow->new;

my $lf = LoginForm->new($mw, 
   -logincmd => \&AuthService::login
)->pack;
my $msg = $lf->Subwidget('Message');

$mw->update;
SetForegroundWindow(oct($mw->id));

#####################################
## No Parameters Set
#####################################
simulateLogin($lf, "", "");
is($msg->cget(-text), "Login Failed!",
   "No user or password set");

#####################################
## Good credentials
#####################################
simulateLogin($lf, "eric", "swordfish");
is($msg->cget(-text), "Login Successful",
   "Valid credentials set");

#####################################
## No username set, good password
#####################################
simulateLogin($lf, "", "swordfish");
is($msg->cget(-text), "Login Failed!",
   "User not set, Password set");

#####################################
## Correct User set, no password
#####################################
simulateLogin($lf, "eric", "");
is($msg->cget(-text), "Login Failed!",
   "User set, no password");
            

sub simulateLogin
{
   my ($lf, $user, $pass) = @_;

   clearLoginForm($lf);
   my $userH = oct($lf->Subwidget('UserEntry')->id);
   my $passH = oct($lf->Subwidget('PassEntry')->id);
   my $butH = oct($lf->Subwidget('LoginButton')->id);

   SendKeys("{TAB}"); 
   SendKeys($user) if $user ne "";
   SendKeys("{TAB}");
   SendKeys($pass) if $user ne "";
   SendKeys("{TAB}");
   SendKeys("{ENTER}");
   $lf->update;
 }

sub clearLoginForm
{
   my $lf = shift;
   $lf->Subwidget('Message')->configure(-text => "");
   $lf->Subwidget('UserEntry')->delete(0, 'end');
   $lf->Subwidget('PassEntry')->delete(0, 'end');
}
[download]

Each time I attempt to test using the toolkit, or the window manager, I learn something new, or run into some problem that I didn't expect. This time, the biggest surprise, was that attempts to trigger the button by using eventGenerate('<ButtonPress-1>') for Tk, and PushButton(HWND) for Win32::GuiTest.

With eventGenerate I was able to make the Button appear to be depressed then released, but for whatever reason, it did not trigger the bindings! (I'm going to have to investigate this later). I was equally unsuccessful using PushButton. It didn't even seem to do anything, however this might have been because the change in Button state occurred faster than I could follow.

These sorts of problems are demonstrative of some of the pain I mentioned earlier. Some of these issues could have more to do with Tk's underlying implementation. Under the hood, it's possible that Tk isn't using a real Win32 Button and is faking it. At any rate, it's issues like these that make a few of BrowserUk's points for him. ;-).

Update: I thought that the approach using {ENTER} with Win32::GuiTest perfectly valid, because it is a binding that should be tested, however a better way of testing the Button would have been using SendMouse("{LEFTCLICK}") after position the pointer above the button with MouseMoveAbsPix. The x/y coordinates are easy enough to fetch either through GuiTest, or using $widget->rootx, $widget->rooty

Closing Thoughts

In this mammoth posting I've shown various ways that a GUI can be tested programmatically. The code I presented was trivial, but the basic concepts can be applied to much larger problems. I have successfully use this approach in some fairly large GUI applications, where sections of the GUI were decomposed into several modules.

I'd like to reemphasize that this style of testing is helpful more for Developers to check to see if their code is testing some fairly predictable pathways, and not a substitute for thorough testing done using other approaches. Usability testing and adherence to GUI standards guidelines are not addressed by these techniques

Finally... Thanks to BrowserUk for a couple of great posts on the subject and helpful offline feedback. I also appreciated comments from zentara as I drafted this. I think I'll clean it up a bit more and repost this on www.perltk.org.

Comments, criticism, and other feedback welcome.

Rob Seegel (RobSeegel@comcast.net)

Comment on Re: Testing a GUI application Select or Download Code