in reply to Accessing dynamic webpages

I think just adding one more request would do what you need. I assume you are always redirected to the same address just with a different cookie. Untested: 
my $req2 = GET "$url";
$result = $ua->request($req)->as_string;
print $result;

[download]

Update:I forgot about the cookie jar - of course you need that too.

Replies are listed 'Best First'.
ReRe: Accessing dynamic webpages
by pduffy (Initiate) on Jun 17, 2003 at 08:33 UTC
    zby gellyfish,

    Thanks for the insights. Not quite there yet. I think it boils down to the way I'm using the required cookie. Here's the code again:

    1:use LWP::Simple;
    2:use LWP::UserAgent;
    3:use HTTP::Request::Common qw(POST);
    4:use HTTP::Request::Common qw(GET);
    5:use HTTP::Cookies;
    6:$url = 'http://www.biocarta.com/search/gene.asp?';
    7:my $request = POST "$url",
    8:[
    9:ID1 => 48852, 10:]
    11:my $ua = LWP::UserAgent->new;
    12:my $response = $ua->request($request);
    13:print $response->as_string;

    14:'Okay. Let's set up a cookie jar, and extract the cookie 15:my $cjar = HTTP::Cookies->new(file => 'cookies.txt',
    autosave => 1);
    16:$cjar->extract_cookies($response);

    17:'As per your suggestions
    18:my $req2 = GET "$url";
    19:$cjar->add_cookie_header($req2);
    20:$ua->redirect_ok($req2);
    21:$response2 = $ua->request($req2);
    22:print $response2->as_string;

    As is it, it returns the genesrchresults.asp webpage, but without the search results (before, it returned undef). If I use $request on line 21 instead of $req2 it returns a 'Doh! page cannot be displayed error blah blah'

    Any more ideas?

    pDuffy.
[reply]
      You need to add the cookie jar to the user agent: 
      $ua->cookie_jar(file => 'cookies.txt',
autosave => 1);

      [download]
      Do it at the beginning of the code, just after the creation of the user agent, not after the first request. You don't need to extract the cookies or anything - just add the cookie jar that's all. And you don't need the  redirect_ok when you do the redirection manually.

      I would correct your code if you cared to add code tags.

[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom