using Mechanize

alvise has asked for the wisdom of the Perl Monks concerning the following question:

Hi monks, I am not new in writing spiders, but this time I really need your help:

#!/usr/bin/perl
use strict;
use warnings;
use WWW::Mechanize;
use HTTP::Cookies;

my $cookie_jar = HTTP::Cookies->new(file => 'cookies', autosave => 1, 
+ignore_discard => 1);
my $mech = WWW::Mechanize->new( cookie_jar => $cookie_jar, autocheck =
+> 1 );

$mech->get( 'https://oc4jese1ssl.pubblica.istruzione.it/trasparenzaPub
+b/ricercacv.do' );

$mech -> form_id ( 'RicercaCVAF' );

$mech -> select ( "codRegione", "AB_ABRUZZO" );
$mech -> select ( "codProvincia", "CH_CHIETI" );
$mech -> select ( "codComune", "E372_VASTO" );
$mech -> field ( "submitName", "Cerca la scuola" );
$mech -> submit ( );

print $mech -> content;
[download]

the above code does not return the webpage that is returned when the form is manually submitted with those 3 selects selected. What am I doing wrong ? Thanks in advance, Alvise

Comment on using Mechanize Download Code

Replies are listed 'Best First'.
Re: using Mechanize by Corion (Patriarch) on Apr 22, 2016 at 21:43 UTC
The page has Javascript. WWW::Mechanize does not support Javascript. You will need to find out what values your browser sends when you manually submit the information using your browser. See for example the Firefox Live HTTP Headers extension. Send that same data using WWW::Mechanize.	[reply]
Re^2: using Mechanize by runrig (Abbot) on Apr 22, 2016 at 22:00 UTC
Firebug is also an option, especially when Live HTTP Headers doesn't catch all of the requests (I think it was XMLHttpRequests I was having an issue with).	[reply]
Re: using Mechanize by nysus (Parson) on Apr 23, 2016 at 12:32 UTC
You might also want to check out `WWW::Mechanize::Firefox` if you are new to browser automation. This module will actually open up a new browser for you and you can watch it navigate and fill out forms. You don't have to worry about reverse engineering forms with javascript to figure out what they are doing. I know for me that it made browser automation a little easier to figure out because you can actually watch your code in action and see what you did wrong. The downside, of course, is that you can't run this on a machine without a copy of Firefox. To get it working, you need to download the mozrepl add-on. Once installed, you need to turn mozrepl on before it will work with `WWW::Mechanize::Firefox` by going to "Tools" -> "mozrepl" -> "start". By default, mozrepl listens on port 4242 but you can change that under "Tools" -> "mozrepl" -> "change port". I also always do "Tools" -> "Activate on startup". That way my code can fire up it's own copy of FF and start using it all by itself. $PM = "Perl Monk's"; $MCF = "Most Clueless ~~Friar~~ ~~Abbot~~ ~~Bishop~~ ~~Pontiff~~ ~~Deacon~~ Curate"; $nysus = $PM . ' ' . $MCF; Click here if you love Perl Monks	[reply] [d/l] [select]