in reply to GUI Automation - locate image

I can give a little bit more information to add to what Corion has already said. This will be much faster if you chase it through the windowing system. The downside is that the app will be less portable, but I infer from your question that you're in Windows regardless of where you want to be.

So on to the meat. First, every window on the screen (and some not) in Windows has a Window Handle. This is a long int that you can use to have a conversation with the OS about a window. Given a window's handle, you can access all of the content of the window, including the title, any text fields, and any buttons. (This is a handy way to send a 'click' message to a button, and it's like the user clicked it.)

Once the Perl app has the process ID, it can wander the list of windows for that process, start asking for handles, and examining window titles. (Titles are good for error messages, because they're usually consistent even when the error message itself varies.)

Some things that might help:

The people at http://sysinternals.com/ write really useful Windows utilities. "Handle" and "Process Explorer" might help here.

Microsoft publishes all of their APIs on the web, you just have to dig to find them. Start with http://msdn.windows.com, and choose library. The API calls that will get you started are FindWindow (Find a top-level window) FindWindowEx (find a child window) and GetWindowText.

This will not result in pretty or interesting code, but it will be so much faster than a graphics approach.

I'd post a sample, but all of my hacking on this was done a few years ago, for a former employer, in Visual Basic. :(

--
Spring: Forces, Coiled Again!

Replies are listed 'Best First'.
Re: Re: GUI Automation - locate image
by Corion (Patriarch) on Mar 09, 2004 at 08:22 UTC

    I'm at work now, where I use Win32::GUITest to automate a stubborn application, and here is a modified version of the spy-- program that I used to look at the current selection in a listbox. The stubborn program displays its status by filling a listbox with all possible status values and then selecting the active status...

    # $Id: guitest-spy.html,v 1.1 2001/09/19 23:31:36 erngui Exp $
#
# MS has a very nice tool (Spy++).
# This is Spy--
#
use Win32::GuiTest qw/FindWindowLike GetWindowText GetClassName IsWind
+owEnabled GetWindowID
    GetChildDepth GetDesktopWindow GetComboContents GetListContents Ge
+tListText SendKeys /;
use Win32::API;

for (FindWindowLike(0,qr/^Explorer -/)) {
    SendKeys("%ao");
    print "Looking for dialog\n";
    my @dialog = FindWindowLike($_,qr/Optionen/,"");
    print "Looking for listbox\n";
    my @listbox = FindWindowLike($dialog[0],"","",752);

    while (1) {
      print "Getting current selection\n";
      my $i = SendMessage($listbox[0], LB_GETCURRSEL,0,0 );
      print $i;
      print "Getting current selection text\n";
      my $t = GetListText($listbox[0],$i);
      print $t,"\n";
      sleep 1;
    };
}

    [download]
[reply]
[d/l]
      Corion, ++ and thanks for the example! We're hoping to start writing some GUI test automation here, could I steal^H^H^H^H^H copy that code for our internal use? It would sure be nice to be able to develop in-house what CA wanted $20K+ for, and that code will save me at least a few hours.

      --
      Spring: Forces, Coiled Again!
[reply]
Re: Re: GUI Automation - locate image
by Anonymous Monk on Mar 09, 2004 at 22:49 UTC
    Thanks, but Application is running within a windows frame utilizing some Java SunAwtCanvas... anything within it is not visible to me from Perl. only graphically. Found the Win32::ActAcc module, here is the Perl code I used to detect any "lasting change" over a 2 second period... It appears very handy for working with other applications... 
    use Strict;
use Win32::OLE;
use Win32::ActAcc;
use Win32::GuiTest qw(:ALL :SW);
 
 my @win;
 my %Wnm;
 my %VW;

 Win32::OLE->Initialize();
 while ( 1 ) {
  @win = ();
  @win = FindWindowLike(); 

  #*# report any "lost" windows:
  for $win ( @win ) { delete $Wnm{$win}; }

  if ( scalar keys %Wnm) {
    print "Lost:\n";
    my @Lost = sort keys %Wnm;
    for (@Lost) { print "$Wnm{$_}\n"; delete $Wnm{$_}; }
    print "\n";
  }

  for $win ( @win ) { dumpwin($win); }

  print "\n----\n";
  sleep(2);
 }
exit;

sub dumpwin {
  my $win = shift;
  print "Null handle\n", return unless ($win); 

  my $w_text   = $win . ' ' . GetWindowText($win); 
     $w_text  .= " class="  . GetClassName($win);
  my $w_state  = ' V=' . IsWindowVisible($win) . ' E='; 
     $w_state .= IsWindowEnabled($win); 
  $Wnm{$win}  = $w_text . ' ' . $w_state;

  my $ao = Win32::ActAcc::AccessibleObjectFromWindow($win);
  my $aatxt = '';
  $aatxt = $ao->describe() if defined $ao;
  $w_state .= ' ' . $aatxt;

  if ( exists $VW{$win} ) {  
    return if ( $VW{$win} eq $w_state );
  }
  $VW{$win} = $w_state;

  print $win,      GetWindowText($win);
  print " class=", GetClassName($win);
  print " vis=",   IsWindowVisible($win); 
  print " en=",    IsWindowEnabled($win); 
  print "\n ",$aatxt, "\n"; 
}

    [download]
[reply]
[d/l]

In Section Seekers of Perl Wisdom