in reply to LWP & Javascript

One particular webpage will not let me grab it because it insists that I need Javascript enabled in order to view the page.

You're going to need to reverse engineer the mechanism by which the server is determining that you don't have Javascript enabled. It shouldn't be too difficult. The page immediately beforehand is probably trying to do something like setting a cookie from withing a <script> block.

Once you know, you'll be able to work around it.

Replies are listed 'Best First'.
Re: Re: LWP & Javascript
by belg4mit (Prior) on Jul 28, 2002 at 23:56 UTC
    Another, possibly more common, and naive, mechanism would be to simply sniff the user agent and compare to a browscap (Browser Capabilities) database. In this case passing the user agent header from a known JavaScript capable browser ought to work.

    --
    perl -pew "s/\b;([mnst])/'$1/g"

[reply]
Re: Re: LWP & Javascript
by Anonymous Monk on Jul 25, 2002 at 05:52 UTC
    Thanks for the reply dws but no dice. I'm already accepting cookies: 
    $ua->cookie_jar(HTTP::Cookies->new(file => "cookies.yum",
                                   autosave => 1));

    [download]
    Here's some more detail. I'm attempting to submit a form and "scrape" the results. Problem being that the webpage uses Javascript form validation and even uses Javascript to do the "submit". I'm really scratching my head here. Here's a snippet of the actual code: 
    my $queryStr = "http://track.airborne.com/TrackByNbr.asp?txtTrackNbrs=
+$trackNo&hdnTrackMode=nbr&hdnPostType=init&hdnRefPage=0&hdnSent=false
+";

my $ua    = new LWP::UserAgent;
my $agent = "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98)";

$ua->agent($agent);
$ua->cookie_jar(HTTP::Cookies->new(file => "cookies.yum",
                       autosave => 1));

my $req = new HTTP::Request POST=>$queryStr;
$req->content_type('application/x-www-form-urlencoded');

my $res = $ua->request($req);

if ($res->is_success) {
    my $response = $res->content;
    open (HTMLDUMP, ">$HTMLFile") ||
        die "Could not create $HTMLFile - $!\n";

    print HTMLDUMP $response;
    close (HTMLDUMP);
} else {
    die "Something bad happened...\n";
}

    [download]
[reply]
[d/l]
[select]
      no dice. I'm already accepting cookies

      Yes, but unless you're executing the Javascript that comes back with a page, you might not be sending cookies that are created from within <script> tags. That is a standard technique that sites use to determine whether the browser had Javascript enabled. A cookie jar isn't going to do you any good in this case.

[reply]
        I'm going to show my ignorance here so please bear with me. How would I execute said Javascript using LWP::*?

        Thanks.
[reply]
[d/l]

In Section Seekers of Perl Wisdom