why does my as_string fail the second time its used

idova has asked for the wisdom of the Perl Monks concerning the following question:

    my $h = new HTTP::Headers(
        Accept=>'*/*',
        user_agent=>$ENV{'HTTP_USER_AGENT'},
    );

    &progress_log("setup the headers");

    my $req = new HTTP::Request(
        $req_type,
        $posttourl,
        $h
    );
    &progress_log("setup the headers");

    my $user_page = $ua->request($req);
    &progress_log("got the results");
    $user_page_flat = $user_page->as_string;
[download]

the code above executes cleanly the first time trough (it runs in a deamon) but the second time (about 24 hours later) the last line fails (the progress_log subs just prints the message to a log so i was able to identify the failing line) now can anyone give me any suggestions as to why it is happening and how to fix it

Comment on why does my as_string fail the second time its used Download Code

Replies are listed 'Best First'.
Re: why does my as_string fail the second time its used by Corion (Patriarch) on Nov 27, 2003 at 10:37 UTC
You don't tell us in what way the `as_string` method "fails", but from your description of locating the offending line, I assume that your program stops executing there. My guess is, that it is not the `as_string` mehtod that actually fails, but simply that `$user_page` is not what you believe it to be - most likely, it's undefined, as `$ua->request($req)` returned `undef` because of some failure. All I can recommend to you is more thorough error checking and logging all output (especially all the output to STDERR) to a logfile, possibly by reopening STDOUT and STDERR to files, or setting up $SIG{__WARN__} and $SIG{__DIE__} handlers that call `progress_log` : `sub log_die { progress_log( '***' . join ' ', @_ ); }; sub log_warn { progress_log( '???' . join ' ', @_ ); }; BEGIN{ $SIG{__WARN__} = \&log_warn; $SIG{__DIE__} = \&log_die; };` [download] Of course, you might also be interested in switching from manipulating LWP requests to using WWW::Mechanize, which tries harder to be just like any browser. `perl -MHTTP::Daemon -MHTTP::Response -MLWP::Simple -e ' ; # The $d = new HTTP::Daemon and fork and getprint $d->url and exit;#spider ($c = $d->accept())->get_request(); $c->send_response( new #in the HTTP::Response(200,$_,$_,qq(Just another Perl hacker\n))); ' # web` [download]	[reply] [d/l] [select]

Replies are listed 'Best First'.

Re: why does my as_string fail the second time its used
by Corion (Patriarch) on Nov 27, 2003 at 10:37 UTC

You don't tell us in what way the as_string method "fails", but from your description of locating the offending line, I assume that your program stops executing there.

My guess is, that it is not the as_string mehtod that actually fails, but simply that $user_page is not what you believe it to be - most likely, it's undefined, as $ua->request($req) returned undef because of some failure.

All I can recommend to you is more thorough error checking and logging all output (especially all the output to STDERR) to a logfile, possibly by reopening STDOUT and STDERR to files, or setting up $SIG{__WARN__} and $SIG{__DIE__} handlers that call progress_log :

sub log_die {
  progress_log( '***' . join ' ', @_ );
};
sub log_warn {
  progress_log( '???' . join ' ', @_ );
};
BEGIN{
  $SIG{__WARN__} = \&log_warn;
  $SIG{__DIE__} = \&log_die;
};
[download]

Of course, you might also be interested in switching from manipulating LWP requests to using WWW::Mechanize, which tries harder to be just like any browser.

perl -MHTTP::Daemon -MHTTP::Response -MLWP::Simple -e ' ;    # The  
$d = new HTTP::Daemon and fork and getprint $d->url and exit;#spider
($c = $d->accept())->get_request(); $c->send_response( new   #in the
HTTP::Response(200,$_,$_,qq(Just another Perl hacker\n))); ' #  web
[download]

[reply]
[d/l]
[select]