LWP::Simple is just that. Simple. It will hang if it can open a socket to a server but the server then plays dead and does not respond. LWP::UserAgent is much more sophisticated with its timeouts and will return if it times out.......

BUT THERE IS A CAVEAT: Because of the way it is implemented in LWP::UserAgent the timeout is best termed as *fuzzy*. It will timeout but if you have a connection to a (say) overloaded server that trickle feeds data back to your client application it may take orders of magnitude longer to timeout than the number you specify. If you absolutely need a hard limit use alarm.

Here is an example. Although the timeout is set to 2 seconds LWP will wait 30 seconds to get the trickle data. Effectively what the timeout means to LWP is 1) make the connection within TIMEOUT seconds and 2) get at least one byte of data from a non blocking socket every TIMEOUT seconds. It will return if either of these are not fulfilled but as you can see you can still kinda hang it.

As an aside it you are scraping sites, and they run anti-scraping response throttling this scenario if quite realistic. 

# this is the test server that does a tricke feed response

#!/usr/bin/perl
use IO::Socket;
use IO::Select;
$lsn = IO::Socket::INET->new( Listen    => 1,
                        LocalAddr => 'localhost',
                        LocalPort => 9000,);

my $client = new IO::Select( $lsn );

while( my @ready = $client->can_read ) {
    for my $fh (@ready) {
        if($fh == $lsn) {
            warn "Accepted new socket\n";
            my $new = $lsn->accept;
            $client->add($new);
        }
        else {
            # Process socket
            warn "Getting data\n";
            $data = <$fh>;  # yeah yeah this is only a toy app
            warn "Got $data\nDoing stuff slowly!\n";
            my @response = split '', "HTTP/1.1 200 OK\n\nHello World!\
+n";
            for( @response) {
                warn $_;
                sleep 1;
                print {$fh} $_;
            }
            $client->remove($fh);
            $fh->close(); 
        }
    }
}
   

# this is the test client

#!/usr/bin/perl
use LWP::UserAgent;
use Data::Dumper;
my $ua = LWP::UserAgent->new( timeout => 2 );
$response = $ua->get('http://localhost:9000/');
print Dumper $response;

[download]

cheers

tachyon

In reply to Re: Get Timout with LWP by tachyon
in thread Get Timout with LWP by ecuguru

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.