Vinzzz has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks,
(Little attention : I'm French my English should be good, but I know sure :-/ I try I try ! :D )
I got a little problem with an Automatic Web surfing script that I try to make.
I have to get an HTML report. This Report is a standard Html Table. To get this report I need to authentify myself with an Input forms, and go to the reports page.
For this part, everything is good. I use an UserAgent, cookies ..etc.

BUT (hehehehe yes here is the "BUT")
The HTML reports "need" some time to show data. First We have the message "Please waiting, we extract your data and make the report table ...etc" Here is my code : 

my $req = HTTP::Request->new(GET=>"$page"); 
my $res = $ua->request($req); #UA is the UserAgent 
die $res->status_line if not $res->is_success; 
my $contentpage= $res->content(); 
open (DEST,">rapport.html"); 
print DEST $contentpage;

[download]

I get the html page with "Please waiting for ...etc" But not the final page with the report ^^ !!
I think that my request takes the first "screenshot" page, but not waiting for the transaction ending.
My Dear monks, I need your great wisdom, I'm a little padawan for sure :D !

Replies are listed 'Best First'.
Re: HTTP::request How "wait" ?
by polettix (Vicar) on May 23, 2005 at 09:40 UTC
    The interim page you're getting is likely to have some refresh timer that instructs the browser to reload the page after some time, just to give the server time to perform the evaluation. I'm saying "reload" but you could actually be redirected somewhere else, just look inside the META tags at the beginning of the interim page, you'd find something related to it.

    For this kind of automated stuff, however, a look to WWW::Mechanize is worth the time, even if I don't know if it's able to automate this aspect as well.

    Update: more on http-equiv and refresh here.

    Flavio (perl -e 'print(scalar(reverse("\nti.xittelop\@oivalf")))')

    Don't fool yourself.
[reply]
[d/l]
[select]
      Thanks for your two answers, I'll read documents and try to give some milestones in this projetcs ;) This is an interim page, sure. The server create a report page for each user request. In the Interim page, it's Javascript code. With a Bar Progress. When the Bar Progress ends, the interim page load the report page. I have to read if the associated URL is a specific one or not. If I get the answer, I'll post my code :) Thanks guys :) Vinzzz - Litle Perl Padawan -
[reply]
Re: HTTP::request How "wait" ?
by awohld (Hermit) on May 24, 2005 at 16:43 UTC
    Try this:
    #!/usr/bin/perl -w

use LWP;
use strict;

my $page = "http://www.site.com";

my $browser = LWP::UserAgent->new();   # Create virtual browser
$browser->cookie_jar( {} );            # Enable cookies

my $response = $browser->get($page);
die "Error: ", $response->status_line
 unless $response->is_success;

my $contentpage = $response->content();

open (DEST, ">rapport.html");
print DEST $contentpage;

    [download]
[reply]
[d/l]
Re: HTTP::request How "wait" ?
by ghenry (Vicar) on May 23, 2005 at 09:56 UTC
[reply]
[d/l]

      I downvoted the parent node because I think it has absolutely nothing to do with the problem. If you don't know the answer, please don't throw out random ideas hoping to solve the problem -- at least not without warning people that you're only guessing.

[reply]

Back to Seekers of Perl Wisdom