The age old JavaScript WWW::Mech scenario

uni_j has asked for the wisdom of the Perl Monks concerning the following question:

Hey guys, trying to debug this script. It's not returning a result and I suspect it is because there is a JavaScript that executes. I checked the header info with live headers and tried to manually post the url : http://www.utsc.utoronto.ca/~registrar/scheduling/timetable?sess=summer&course=CSC&submit=Display+by+Discipline&course2= But the table does not build the list. I tried looking at the javascript and couldn't make sense of how I can trickit to work with mech. Here is my code, I hope someone has a good idea of how I can get around this problem.

use WWW::Mechanize;
use HTML::TokeParser;
my $mech = WWW::Mechanize->new(cookie_jar => undef);
$url="http://www.utsc.utoronto.ca/~registrar/scheduling/timetable";
$prefix = "ANT";
$mech->get($url);

$mech->submit_form(
    fields =>{
        sess => "summer",
        course => $prefix,
        },
    );

print $mech->content();
[download]

Comment on The age old JavaScript WWW::Mech scenario Select or Download Code

Replies are listed 'Best First'.
Re: The age old JavaScript WWW::Mech scenario by Corion (Patriarch) on Apr 29, 2009 at 17:40 UTC
If you want to find out whether a site uses JavaScript for essential functions, just disable Javascript in your browser and visit the site. If the page still renders well (enough), it doesn't need Javascript and WWW::Mechanize will Just Work. If you think you really need Javascript, consider looking at WWW::Scripter to get WWW::Mechanize with a built-in Javascript engine.	[reply]
Re: The age old JavaScript WWW::Mech scenario by perrin (Chancellor) on Apr 29, 2009 at 18:41 UTC
There is no way for JavaScript to do anything that you can't imitate with Mech talking to the server. It can only speak HTTP, so all you have to do is look at what it sends and send the same thing. You were on the right track with LiveHTTPHeaders. You probably missed something, or need a cookie, or need to send it as POST instead of GET, etc.	[reply]
Re: The age old JavaScript WWW::Mech scenario by whakka (Hermit) on Apr 29, 2009 at 21:44 UTC
Without specifying which, mech automatically selects the first form; you need to select the second form on that url, so: `$mech->submit_form( form_number => 2, fields =>{ sess => "summer", course => $prefix, }, );` [download] gives me something. UPDATE:...and that something wasn't good. I've always found it easier to debug form automation in `WWW::Mechanize` by doing one thing at a time. This works for me: `use WWW::Mechanize; use HTML::TokeParser; my $mech = WWW::Mechanize->new(cookie_jar => undef); $url="http://www.utsc.utoronto.ca/~registrar/scheduling/timetable"; $prefix = "ANT"; $mech->get($url); $mech->form_number(2); $mech->set_visible( [radio => 'summer'] ); # page default anyway $mech->select( 'course', $prefix ); $mech->click_button( name => 'submit' ); print $mech->content();` [download] - clarified first sentence.	[reply] [d/l] [select]
Re^2: The age old JavaScript WWW::Mech scenario by trwww (Priest) on Apr 30, 2009 at 00:39 UTC
UPDATE:...and that something wasn't good. You're probably just missing the button that needs clicked to send the form: `$mech->submit_form( form_number => 2, fields =>{ sess => "summer", course => $prefix, }, button => 'submit', );` [download] That might work.	[reply] [d/l]
Re^3: The age old JavaScript WWW::Mech scenario by whakka (Hermit) on Apr 30, 2009 at 01:29 UTC
Yep, that's probably more elegant. TMTOWTDI - to each his own :)	[reply]
Re: The age old JavaScript WWW::Mech scenario by imrags (Monk) on Apr 30, 2009 at 08:04 UTC
To check script and other code in html, i like using firefox with "firebug". That gives me all the necessary things i want to know about a page. WWW::Mechanize does not support javascript as monks have already mentioned. There are a few modules available. "Super Search" will guide you to the same. Raghu	[reply]