larard has asked for the wisdom of the Perl Monks concerning the following question:

I have been wanting to scrape a site using WWW::Mechanize::Firefox. I can successfully fill in the first form, but the second one seems to have a strange submit button that I am unable to click. 
use strict;
use warnings;
use local::lib;
use WWW::Mechanize::Firefox;

my $mech = WWW::Mechanize::Firefox->new( autoclose => 0 );
$mech->get('http://www.rightmove.co.uk/');
# Filling in and submitting the first forms works:
$mech->submit_form(
                   with_fields => {
                                   searchLocation => 'N1',
                                  }
                  );
# The second form I can't work out....
$mech->submit_form({with_fields => {'sortByPriceDescending'=>''}});
#This was the otherway I thought of doing it, 
#$mech->click({xpath => '//*[@id="submit"]'});

[download]
I know that I could just work out the url scheme for the results page and go there directly, but I was trying to do it the 'correct' way, and have all the javascript run, etc.

In short how do I "click" the 'Find properties" button on the second page?

Replies are listed 'Best First'.
Re: Using WWW::Mechanize::Firefox to navigate a site
by Corion (Patriarch) on Aug 17, 2010 at 20:00 UTC

    Your second way should work:

    $mech->click({xpath => '//*[@id="submit"]'});

    [download]

    Alternatively, split up the action into two steps to further investigate:

    my $find_button = $mech->xpath('//*[@id="submit"]', single => 1 );
$mech->click( $find_button );

    [download]
[reply]
[d/l]
[select]
      Unfortunately neither of those work for me. (But I am glad that I read the docs enough that you, the author of this excellent module, suggest it should work).

      Also, I think you're saying that there should be more debugging output from the 2 step method you suggest. There is, which I believe suggests that the xpath is wrong (I got it from firebug's inspect element followed by copy xpath). The error I got is: 

      
No elements found for '//*[@id="submit"]' at scrape line 15
No elements found for Button with name '' at scrape line 16
Can't call method "__click" on an undefined value at /home/tom/perl5/l
+ib/perl5/WWW/Mechanize/Firefox.pm line 1924, <DATA> line 1.

      [download]
[reply]
[d/l]

        The button seems to be added only "later" on, after even DOMContentLoaded has fired. I get your script to work consistently by using sleep 1 to give the page more time to construct itself:

        use strict;
use warnings;
#use local::lib;
use WWW::Mechanize::Firefox;
warn $WWW::Mechanize::Firefox::VERSION;

my $mech = WWW::Mechanize::Firefox->new(
    autoclose => 0,
    autodie => 1,
);
$mech->get('http://www.rightmove.co.uk/');
# Filling in and submitting the first forms works:
$mech->submit_form(
                   with_fields => {
                                   searchLocation => 'N1',
                                  }
                  );

sleep 1; # give the page more time
$mech->click({xpath => '//*[@id="submit"]'});

        [download]

        To make this truely robust, you would need to wait (always with a timeout!) until the search button becomes visible:

        my $retries = 10;
while ($retries-- and ! $mech->is_visible( xpath => '//*[@id="submit"]
+' )) {
      sleep 1;
};
die "Timeout" unless $retries;

        [download]
[reply]
[d/l]
[select]

      Dear Corion, I have a doubt on this thread. Can you pls suggest a way?

      Is there a provision to pass a string to it like below,

       my $Pager = "submit";

my $find_button = $mech->xpath('//*[@id="$Pager"]', single => 1 );
$mech->click( $find_button );

      [download]

      Its not getting me through! Giving an error like 'NO elements found'

      Thanks in advance.

[reply]
[d/l]

Back to Seekers of Perl Wisdom