smetj has asked for the wisdom of the Perl Monks concerning the following question:

Hi I have a web application which needs to be polled by a perl script. This perl script will be used as a plugin in Nagios so it has to perform as fast as possible.
The http request itself is just a simple http post.
My first choice would be LWP::UserAgent, but it appears to be quite slow. I compared the speed of LWP::UserAgent by letting my Perl script use /usr/bin/curl binary instead, which is to my suprize a lot faster.

Query using LWP::UserAgent 
#!/usr/bin/perl
use strict;
use warnings;
use LWP::UserAgent;
use JSON;

my $client = new LWP::UserAgent;
my $json = new JSON;

my %request=('commandlet'=>'memory','output'=>'json');
my $json_request = $json->encode(\%request);

my $web_request = $client->post('http://localhost:12345/query',{ usern
+ame => 'default',
                                password => 'changeme',
                                json => $json_request
                                });
print $web_request->content;

[download]
Query using Curl: 
#!/usr/bin/perl
use strict;
use warnings;
use JSON;
my $json = new JSON;
my %request=('commandlet'=>'memory','output'=>'json');

my $json_data=$json->encode(\%request);
my $command=sprintf"/usr/bin/curl -d username='%s' -d password='%s' --
+data-urlencode json='%s' http://localhost:12345/query 2>/dev/null|","
+default","changeme",$json_data; 

open FH,$command;
while (<FH>){
    print $_;
    }
close FH;

[download]
If I write a bash script which iterates 100 times over both scripts I get following results: 
Using curl:

real    0m4.006s
user    0m3.060s
sys    0m0.740s

Using LWP

real    0m12.978s
user    0m11.405s
sys    0m1.184s

[download]
I'm running Ubuntu 9.1 and LWP::UserAgent has version "5.829"
Has anyone an idea why LWP::UserAgent is so slow, or is this rather normal?
How could I speed up a simple post like this without the need to use the external binary curl all the time?
Thanks,
Jelle

Replies are listed 'Best First'.
Re: Why is LWP::UserAgent so slow?
by zwon (Abbot) on Feb 10, 2010 at 19:38 UTC
    If I write a bash script which iterates 100 times over both scripts I get following results:

    You comparing not just the times of sending request, but also the times of scripts compiling. First script imports LWP and JSON, and second only JSON. Look onto my test results: 

    $ time (for i in `seq 1 100`; do perl -MJSON -e1; done)

real    0m2.635s
user    0m1.930s
sys    0m0.510s

$ time (for i in `seq 1 100`; do perl -MJSON -MLWP::UserAgent -e1; don
+e)

real    0m7.111s
user    0m6.060s
sys    0m0.740s

    [download]
[reply]
[d/l]

      True but LWP::UserAgent is slower than some other methods probably (as already said) down to its flexibility. Not that I am recommending HTTP::Lite (in fact HTTP::Lite pod says use LWP) but to retrieve a file on our local network with LWP::UserAgent and HTTP::Lite where I know it is UTF-8 encoded (so simplified) I get:

      use 5.010;
use strict;
use warnings;
use Benchmark;
use Encode;

use HTTP::Lite;
use LWP::UserAgent;
my $url = 'xxx.yyy.zzz/file.dat';

sub http_lite {
    my $r = HTTP::Lite->new;
    my $sts = $r->request($url);
    die "failed" if $sts != 200;

    my $decoded = Encode::decode('utf8', $r->body, Encode::FB_CROAK);
}

sub lwp_useragent{

    my $ua = LWP::UserAgent->new;
    my $r = $ua->get($url);
    die "failed" if !$r->is_success;
    my $body = $r->decoded_content;
}

timethese(1000, {
    'lite' => sub{http_lite()},
    'lwp' => sub{lwp_useragent()}
   });


Benchmark: timing 1000 iterations of lite, lwp...
      lite:  3 wallclock secs ( 1.53 usr +  0.60 sys =  2.13 CPU) @ 46
+9.48/s (n=1000)
       lwp:  6 wallclock secs ( 4.19 usr +  0.58 sys =  4.77 CPU) @ 20
+9.64/s (n=1000)

      [download]

      As I said, I'm not recommending HTTP::Lite just pointing out the difference in a simple case. It does not do HTTPS and defaults to HTTP 1.0 etc and maybe there is something else I've forgotten in the above example that LWP::UserAgent is doing that the HTTP::Lite sub is not.

[reply]
[d/l]
      Yep indeed, I didn't take the compile time into account. The larger the modules, the more time it takes obviously.. thanks
[reply]
Re: Why is LWP::UserAgent so slow?
by Anonymous Monk on Feb 10, 2010 at 10:35 UTC
    because it is so flexible
[reply]

Back to Seekers of Perl Wisdom