in reply to Re: LWP & Javascript
in thread LWP & Javascript

Thanks for the reply dws but no dice. I'm already accepting cookies: 
$ua->cookie_jar(HTTP::Cookies->new(file => "cookies.yum",
                                   autosave => 1));

[download]
Here's some more detail. I'm attempting to submit a form and "scrape" the results. Problem being that the webpage uses Javascript form validation and even uses Javascript to do the "submit". I'm really scratching my head here. Here's a snippet of the actual code: 
my $queryStr = "http://track.airborne.com/TrackByNbr.asp?txtTrackNbrs=
+$trackNo&hdnTrackMode=nbr&hdnPostType=init&hdnRefPage=0&hdnSent=false
+";

my $ua    = new LWP::UserAgent;
my $agent = "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98)";

$ua->agent($agent);
$ua->cookie_jar(HTTP::Cookies->new(file => "cookies.yum",
                       autosave => 1));

my $req = new HTTP::Request POST=>$queryStr;
$req->content_type('application/x-www-form-urlencoded');

my $res = $ua->request($req);

if ($res->is_success) {
    my $response = $res->content;
    open (HTMLDUMP, ">$HTMLFile") ||
        die "Could not create $HTMLFile - $!\n";

    print HTMLDUMP $response;
    close (HTMLDUMP);
} else {
    die "Something bad happened...\n";
}

[download]

Replies are listed 'Best First'.
Re: Re: Re: LWP & Javascript
by dws (Chancellor) on Jul 25, 2002 at 06:01 UTC
    no dice. I'm already accepting cookies

    Yes, but unless you're executing the Javascript that comes back with a page, you might not be sending cookies that are created from within <script> tags. That is a standard technique that sites use to determine whether the browser had Javascript enabled. A cookie jar isn't going to do you any good in this case.

[reply]
      I'm going to show my ignorance here so please bear with me. How would I execute said Javascript using LWP::*?

      Thanks.
[reply]
[d/l]
        How would I execute said Javascript using LWP::*?

        You wouldn't. That's why I'm suggesting that you reverse engineer what's doing on in the page. Once you understand what the Javascript detection mechanism is, you can work around it (via some clever regexs, or whatever) without having to simulate Javascript execution.

[reply]
        You can't. HTML is parsed, Javascript is executed.

        LWP does not contain any Javascript engine that I'm aware of. You might try using pavuk though, which does support Javascript, IIRC.

[reply]
        Not with LWP. But you can with JavaScript. OTOH if this cookie setting bit is the problem you'd be better off just parsing the JavaScript in your code; you don't actually want to *execute* JavaScript, just emulate it's results.

        --
        perl -pew "s/\b;([mnst])/'$1/g"

[reply]

In Section Seekers of Perl Wisdom