davemcgi has asked for the wisdom of the Perl Monks concerning the following question:

Okay here's the code snippet first ...
    $UserAgent = new LWP::UserAgent;
    $UserAgent->timeout(10);
    $UserAgent->agent("Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;
+ rv:1.7.5) Gecko/20041107 Firefox/1.0");
    
    $MyRequest = HTTP::Request->new('GET',"$URL");
    $MyResponse = $UserAgent->request($MyRequest);

    if ($MyResponse->is_success) {
        $FContent = $MyResponse->content;
        $FCode = $MyResponse->code;
        $FBase = $MyResponse->base;
    } else {
        $FContent = "";
        $FCode = $MyResponse->code;
        $FBase = "";
    }

[download]

And now the problem.

As far as I can tell the "timeout" value only seems to apply to the *connection* itself.

Under heavy usage checking 10,000 external links from our websites, I have seen one or two sites that appear to connect correctly, but then take maybe 5 to 10 minutes to retrieve the page requested.

So it seems once connection is made, the remote server can feed as slowly as he likes, and my script has to wait 10 minutes to retrieve the page.

Is there anyone who has come across a similar problem, and is there any way to wrap the block up in an interrupt or similar, so I can specify "if page takes longer than 30 seconds to transfer, then abort" ?

Thanks in advance

Replies are listed 'Best First'.
Re: LWP Timouts
by virtualsue (Vicar) on Oct 31, 2005 at 12:07 UTC
    Read perldoc -f alarm if you haven't already.

    A quick way to do what you want: 

        
    $MyRequest = HTTP::Request->new('GET',"$URL");
    alarm 30;
    $MyResponse = $UserAgent->request($MyRequest);
    my $timeleft = alarm 0;

    [download]
    Caveats: Too many to list in this space, signal handling has the ability to wildly complicate your life, etc. If your program is very simple I think the above may do what you need. If not, Super Search for examples and ask more questions. :)
[reply]
[d/l]
Re: LWP Timouts
by zentara (Cardinal) on Oct 31, 2005 at 12:45 UTC
    As an alternative to what virtualsue has shown, you could also write you code with an "event-loop" system like POE, or any of the GUI's, like Tk,Gtk2, etc. You could then start separate timers to monitor each lwp request. Another advantage of that type of system, is that you could log the times for each lwp request.
    I'm not really a human, but I play one on earth. flash japh
[reply]
Re: LWP Timouts
by pajout (Curate) on Oct 31, 2005 at 11:42 UTC
    Did you try to use alarm function and related stuff?
[reply]
Re: LWP Timouts
by Anonymous Monk on Oct 31, 2005 at 22:32 UTC
    Look at Sys::SigAction. From the POD: 
              #timeout a system call:
          use Sys::SigAction qw( set_sig_handler );
          eval {
             my $h = set_sig_handler( 'ALRM' ,\&mysubname ,{ mask=>'AL
+RM' ,safe=>1 } );
             alarm(2)
             ... do something you want to timeout
             alarm(0);
          }; #signal handler is reset when $h goes out of scope
          alarm(0);
          if ( $@ ) ...

    [download]
    Sys::SigAction gives you the most reliable signal handling, and therefore timeout handling.
[reply]
[d/l]
Re: LWP Timouts
by davemcgi (Novice) on Nov 01, 2005 at 08:44 UTC
    Thanks to all for your help ... problem solved using SYS:SigAction :-)
[reply]

Back to Seekers of Perl Wisdom