http://www.GIDEONonline.net/authx.php?user=metalib&browser_ok=1
[download]

http://www.GIDEONonline.net/loginx.php?user=metalib
[download]

Generally, if you want to process pages containing JavaScript with LWP::UserAgent, you must carefully read all javascript on the pages you retrieve and decide what workaround to choose in every certain case.

In the worst cases Javascript module is the only solution.

Thanks, it worked like a charm. I didn't exactly get the retrieved page, but I did get a cookie I needed for the search page. Problem solved.

The problem is simply that LWP::UserAgent does not know about and does not handle Javascript. You need to filter out the interesting parts of the Javascript yourself and react accordingly.

Hi. This is an example of a page I retrieved that uses JavaScript. The code is as so:

#!/usr/bin/perl
use strict;
use LWP::UserAgent;
{
    my $ua = new LWP::UserAgent();
    my $search_address = "http://online.wsj.com/search/full.html?";
    #creating the request object
    my $req = new HTTP::Request ('GET', $search_address);
    #sending the request
    my $res = $ua->request($req);
    if (!($res->is_success)){
        warn "Warning:".$res->message."\n";
    }

    my $response = $res->headers_as_string();
    my $response .= $res->content();
    print "$response\n";
}
[download]

If you run this code you should get a response that has Javascripts. As you can see the code is basically the same except for the URL.

How is that different from the content of the page you posted at the top of this thread? That has javascript too.

Are you perhaps confused about <noscript> tags? They don't indicate that no scripts are present, they represent an alternative in case scripts are not executed by the browser. read the w3 specs on SCRIPT and NOSCRIPT tags

"What should it profit a man, if he should win a flame war, yet lose his cool?"

The JavaScript on the page can do many different things. In some cases javascripting can be ignored, in some cases cannot.

In general, if you can work with the page with Javascript turned off in your favorite browser without any loss of essential functionality — you can easily work with the page using LWP::UserAgent.

I have been able to retrieve pages with Javascript in the past. What's different now?

You misunderstand. You are retrieving the page. It's just that no javascript is executed by LWP::UserAgent.
"What should it profit a man, if he should win a flame war, yet lose his cool?"

Since you do not tell us what those past pages looked like (in terms of Javascript), how could we possibly tell you what has changed? We can guess -- perhaps those pages you speak of did not use Javascript for navigational purposes. You could try the Javascript CPAN module. I used this successfully several years ago to decrypt an encrypted web page, but something tells me you might need to parse out the href locations yourself and feed those back to LWP.

jeffa

L-LL-L--L-LL-L--L-LL-L--
-R--R-RR-R--R-RR-R--R-RR
B--B--B--B--B--B--B--B--
H---H---H---H---H---H---
(the triplet paradiddle with high-hat)

If you use Win32::IE::Mechanize instead, that depends on the MSIE core, and it'll respond to Javascript, too. Of course, that'll only work on Windows.

Some time ago I installed JavaScript::SpiderMonkey, but had not played with it so far - so your question is an opportunity ;-)

As others pointed out LWP::UserAgent doesn't parse or evaluate JavaSript. The code below uses JavaScript::SpiderMonkey for that and extracts the JavaScript stuff with HTML::Parser.

This cruft for obvious reasons works only for the link you provided.

--shmem

_($_=" "x(1<<5)."?\n".q·/)Oo.  G°\        /
                              /\_¯/(q    /
----------------------------  \__(m.====·.(_("always off the crowd"))."·
");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}