Jesse Smith has asked for the wisdom of the Perl Monks concerning the following question:

Is there a way to restart LWP::Simple or make it load faster? 
use LWP::Simple;
$ipaddress2 = get ('http://www.domain.com/ip-address.txt');
$number = get ('http://www.domain.com/counter.txt');

[download]
makes the script take about 15 seconds to load vs less than a second with out it. Yet on another domain on the same server, it doesn't slow it down. 
#!/usr/bin/perl

use LWP::Simple;

$ipaddressmain2main = get ('http://www.domain.com/ip-address.txt');
$numbermain = get ('http://www.domain.com/counter.txt');
$ipaddressmain = "/home/site1337/public_html/ip-address.txt";
$countermain = "/home/site1337/public_html/counter.txt";
$database = "/home/site1337/public_html/logs.shtml";
{
  open(numbermain,"$countermain");
  $num = <numbermain>;
  close(numbermain);

  if ($ipaddressmain2main == $ENV{'REMOTE_ADDR'})  { $num; } else { $n
+um++; }

   open(NEWFILE,">$ipaddressmain") || die $!;
   print NEWFILE "$ENV{'REMOTE_ADDR'}\n";
   close(NEWFILE);

   open(NUM,">$countermain") || die $!;
   print NUM "$num";
   close(NUM);
}
$shortdate = `date +"%D %T %Z"`; 
chop ($shortdate);
    open (DATABASE,">>$database");
        print DATABASE "$ENV{'REMOTE_ADDR'} - $shortdate - $ENV{'HTTP_
+USER_AGENT'} - <A HREF=\"http://$ENV{'HTTP_HOST'}$ENV{'REQUEST_URI'}\
+">http://$ENV{'HTTP_HOST'}$ENV{'REQUEST_URI'}</a> - <A HREF=\"$ENV{'H
+TTP_REFERER'}\">$ENV{'HTTP_REFERER'}</a>\n";
    close(DATABASE);

$page = $in{'page'};

if ($page eq 'XXXXXXX') {
   print "Content-type: text/html\n\n";
print <<EOM;

EOM
}

elsif ($page eq 'XXXXX') {
print <<EOM;

EOM
}

else {
use LWP::Simple;
$ipaddress2 = get ("http://www.domain.com/stats/$page/ip-address.txt")
+;
$number = get ("http://www.domain.com/stats/$page/counter.txt");
$ipaddress = "/home/site1337/public_html/stats/$page/ip-address.txt";
$counter = "/home/site1337/public_html/stats/$page/counter.txt";
{
  open(NUMBER,"$counter");
  $num = <NUMBER>;
  close(NUMBER);
  if ($ipaddress2 == $ENV{'REMOTE_ADDR'})  { $num; } else { $num++; }
   open(NEWFILE,">$ipaddress") || die $!;
   print NEWFILE "$ENV{'REMOTE_ADDR'}\n";
   close(NEWFILE);
   open(NUM,">$counter") || die $!;
   print NUM "$num";
   close(NUM);
}
   print "Content-type: text/html\n\n";
print <<EOM;
PAGE CONTENT
EOM
   exit;
}

[download]

Replies are listed 'Best First'.
Re: use LWP::Simple slows script down.
by davido (Cardinal) on Feb 10, 2014 at 05:17 UTC

    LWP::Simple does blocking requests. The big problem you're facing isn't that some requests take longer than others. It is that whatever length of time they take, your script has to sit and wait before it can proceed to the next request. Since you have no control over how long an individual request can take, you might want to restructure your program around an event loop so that it doesn't have to wait before issuing the next request.

    Here is an example of using Mojo::UserAgent and Mojo::IOLoop for non-blocking requests.

    use Mojo::UserAgent;
use Mojo::IOLoop;

my $ua = Mojo::UserAgent->new;

# --- Worst nodes -------------------!
# --- Best nodes --------------!     |
# --- Last hour CB -----!      |     |
#                      \/     \/    \/
foreach my $node ( qw( 596792 9066 9488 ) ) {


  $ua->get( "http://perlmonks.org/?node_id=$node"  =>   sub {

    my( $ua, $mojo ) = @_;

    print $mojo->req->url,  " => <<<\n", 
          $mojo->res->text, "\n>>>\n\n\n";

  } );


}

Mojo::IOLoop->start unless Mojo::IOLoop->is_running;

    [download]

    The point to the exercise is that however many requests you make (within the limits of memory), the IO loop just handles them with the callback you provide as responses come in. Meanwhile, you can continue issuing requests or processing the responses. It's a parallel process, rather than serial.


    Dave

[reply]
[d/l]
      Is it normal for it to be very slow when it's only getting a text file with a number, and a text file with an IP address, with the files on the same domain?
[reply]

        The speed with which a remote server responds to an HTTP request is not within the scope of your local Perl program. The only way that your local program could have an effect on how the server responds would be to know if the remote server uses some metric to selectively throttle requests, and if so, to do your best to program around that remote-servers behavior.

        In other words, in all probability, the remote server is just slow sometimes. However, it's possible that the remote server is detecting your UserAgent by name (or by IP, or by lack of accepting cookies, or some other metric), and selectively choosing to slow down your requests. Either way, it's not an LWP::Simple issue, or even a Perl issue, the remote server is the one who decides how long requests take, not LWP::Simple. If it does turn out that you're being throttled based on some metric, the solution would be to uncover what criteria are resulting in the server slowing down its responses to your requests, and then try to not fall within those criteria. Maybe you need to accept cookies. Maybe you need to rename your user agent. Maybe you need to originate from different IP's, or to slow down the rate at which you're making requests.

        Your best bet, assuming you're living within the TOS of the server you are hitting, is to ask the administrator of that server why his server is slow, and if it's because you're falling into some criteria for the engagement of a safety mechanism, work with the admin to find ways to avoid tripping the behavior.


        Dave

[reply]
Re: use LWP::Simple slows script down.
by karlgoethebier (Abbot) on Feb 10, 2014 at 11:58 UTC
[reply]

Back to Seekers of Perl Wisdom