perl_monster has asked for the wisdom of the Perl Monks concerning the following question:

I have a simple requirement of screen scraping a web-page and direct the HTML response to an output file. The URL will however redirect to an authentication (Login) page with form base authentication (no javascript) and upon authentication the report I am trying to view would show up. Interestingly, my code is working just fine in a Windows machine, however the same code below is not working in AIX machine and it looks like the click_button() function call does nothing. I have tried click(), submit(), but none is working so instead of getting the actual report all I get is the logon screen in the HTML output file. Any ideas, what can be wrong? 

    
use WWW::Mechanize;
use strict;

my $username = "admin";
my $password = "welcome1";    
my $outpath  = "/home/data/output";
my $fromday = 7;
my $url  = "https://www.myreports.com/tax_report.php";
my $name = "tax_report";
my $outfile = "$outpath/$name.html";

my $mech = WWW::Mechanize->new(noproxy =>'0');    
    
my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(t
+ime);
$year += 1900; 
$mon++;     # since it will start from 0
$mday--;     # yesterdays date (to day)
$fromday = $mday - $days; #(from day)
    
#Create URL extension for generating report with previous date
my $dt_range = "?dc_range=Y&dc_date1=$mon%2F$fromday%2F$year&dc_date2=
+$mon%2F$mday%2F$year"; 
my $url  = $url . $dt_range;

$mech->get($url);
$mech->field(login => "$username");
$mech->field(passwd => "$password");
$mech->click_button(value=>"Login now");

my $response = $mech->content();
    
print "Generating report: $name...\n";

open (OUT, ">>$outfile")|| die "Cannot create report file $outfile";
print OUT "$response";
close OUT;

[download]

Replies are listed 'Best First'.
Re: WWW::Mechanize click_button() not working in AIX
by kcott (Archbishop) on Jul 10, 2012 at 16:44 UTC

    You have unterminated statements in your code. Change 

    my $outpath  = "/home/data/output"
my $fromday = 7

    [download]

    to

    my $outpath  = "/home/data/output";
my $fromday = 7;

    [download]

    Consider adding use warnings; near the start of your script.

    -- Ken

[reply]
[d/l]
[select]

      The un-terminated lines of code was a typo here, sorry. The Version of Perl in Win machine is 5.10.1 and the AIX version of Perl is 5.8.8. But if version of perl were the problem then the whole perl Module i.e. WWW::Mechanize should have given errors. I do not see any specific version pre-requisites of perl in the WWW::Mechanize documentation.

[reply]
Re: WWW::Mechanize click_button() not working in AIX
by Corion (Patriarch) on Jul 10, 2012 at 16:44 UTC

    You don't show the HTML, so we can't know what goes wrong. In your place, I would look long and hard at the differences between the two Perl versions on the two machines. It's most likely caused by different module versions on Windows vs. AIX.

[reply]
Re: WWW::Mechanize click_button() not working in AIX
by onelesd (Pilgrim) on Jul 10, 2012 at 17:34 UTC
    Have you tried any of the other available key/value options to target the button with click_button? Possibly you get slightly different HTML when browsing from AIX.
[reply]

      Unfortunately, the only way I can target the button is to use the key/value pair I have used in the code. The HTML is a bit odd, but it belongs to a third party and I have little control over it. It looks like below (posting just the body portion of it which includes the logon form)-

      <body>
    <h1>Account &nbsp;--&nbsp; Log In</h1>
    
    <div id="content">
        <form method="post">
        <table id="login" cellpadding="0" cellspacing="0">
            <tr>
                <th colspan="2" class="right">Log In</th>
            </tr>
            <tr>
                <td class="label">Username:</td>
                <td class="right">
                    <input type="text" name="login" value="" size="15"
+/>                    
                </td>
            </tr>
            <tr>
                <td class="label">Password:</td>                    
                <td class="right">
                    <input type="password" name="passwd" size="15"/>
                </td>
            </tr>
            <tr>
                <th colspan="2" class="last">
                    <input type="submit" value="Login now"/>
                </th>
            </tr>
        </table>
        <input type="hidden" name="action" value="login"/>
        </form> 
   
        <a href="/">Speed Fulfillment & Call Center Home</a>
    </div>
</body>

      [download]

      as you can see that the button is actually a form submit element. So having said this, i have also tried $mech->submit() , but it did not work either.

[reply]
[d/l]
[select]
        Have you compared the perl/module versions like Corion suggested?
[reply]

Back to Seekers of Perl Wisdom