Re: use LWP::Simple slows script down.

LWP::Simple does blocking requests. The big problem you're facing isn't that some requests take longer than others. It is that whatever length of time they take, your script has to sit and wait before it can proceed to the next request. Since you have no control over how long an individual request can take, you might want to restructure your program around an event loop so that it doesn't have to wait before issuing the next request.

Here is an example of using Mojo::UserAgent and Mojo::IOLoop for non-blocking requests.

use Mojo::UserAgent;
use Mojo::IOLoop;

my $ua = Mojo::UserAgent->new;

# --- Worst nodes -------------------!
# --- Best nodes --------------!     |
# --- Last hour CB -----!      |     |
#                      \/     \/    \/
foreach my $node ( qw( 596792 9066 9488 ) ) {


  $ua->get( "http://perlmonks.org/?node_id=$node"  =>   sub {

    my( $ua, $mojo ) = @_;

    print $mojo->req->url,  " => <<<\n", 
          $mojo->res->text, "\n>>>\n\n\n";

  } );


}

Mojo::IOLoop->start unless Mojo::IOLoop->is_running;
[download]

The point to the exercise is that however many requests you make (within the limits of memory), the IO loop just handles them with the callback you provide as responses come in. Meanwhile, you can continue issuing requests or processing the responses. It's a parallel process, rather than serial.

Dave

Comment on Re: use LWP::Simple slows script down. Download Code

Replies are listed 'Best First'.
Re^2: use LWP::Simple slows script down. by Jesse Smith (Acolyte) on Feb 11, 2014 at 18:14 UTC
Is it normal for it to be very slow when it's only getting a text file with a number, and a text file with an IP address, with the files on the same domain?	[reply]
Re^3: use LWP::Simple slows script down. by davido (Cardinal) on Feb 11, 2014 at 19:07 UTC
The speed with which a remote server responds to an HTTP request is not within the scope of your local Perl program. The only way that your local program could have an effect on how the server responds would be to know if the remote server uses some metric to selectively throttle requests, and if so, to do your best to program around that remote-servers behavior. In other words, in all probability, the remote server is just slow sometimes. However, it's possible that the remote server is detecting your UserAgent by name (or by IP, or by lack of accepting cookies, or some other metric), and selectively choosing to slow down your requests. Either way, it's not an LWP::Simple issue, or even a Perl issue, the remote server is the one who decides how long requests take, not LWP::Simple. If it does turn out that you're being throttled based on some metric, the solution would be to uncover what criteria are resulting in the server slowing down its responses to your requests, and then try to not fall within those criteria. Maybe you need to accept cookies. Maybe you need to rename your user agent. Maybe you need to originate from different IP's, or to slow down the rate at which you're making requests. Your best bet, assuming you're living within the TOS of the server you are hitting, is to ask the administrator of that server why his server is slow, and if it's because you're falling into some criteria for the engagement of a safety mechanism, work with the admin to find ways to avoid tripping the behavior. Dave	[reply]