Hi folks.

I have a very specific problem involving Apache::Registry and LWP::UserAgent.

I maintain a website that uses a number of Apache::Registry scripts to access an Oracle database. Every ten minutes or so, an LWP::UserAgent script running on my DSL'ed home PC tries to retrieve an Oracle-sourced dynamic page from the site, and the retrieval time is logged. If it takes longer than usual to retrieve the page, or if an error occurs, then the script emails me.

For the past few days, I've been emailed a number of spurious warnings because UserAgent's request method has been retrieving pages very slowly -- up to 30 or 40 seconds for a 14K page. This slow behavior only happens with the combination of my home PC's LWP::UserAgent and my server's database-driven Apache::Registry scripts.

I checked to see if my scripts were running slowly. According to the time logging I've got on them, they're running in less than a second, typically. Something somewhere is eating up extra time. I don't think it's a network issue, because if I use lynx or links to retrieve the dynamic page, it arrives promptly. If I use wget, though, I see the same slow behavior as with LWP::UserAgent.

If I put a callback routine in the ->retrieve call to print a hashmark every 100 bytes, I notice that the file arrives in chunks, with long intervals between chunks.

If I point my LWP::UserAgent script at other dynamic sites, like slashdot, those sites load promptly.

I set up a short cgi program on my server to print out 14K of random garbage to the browser. This page loads in an appropriate interval using LWP::UserAgent. I also tried retrieving a static document of comparable size from the server using LWP::UserAgent, and the performance was fine.

So, I'm kinda flummoxed. Does anyone know of any pitfalls in LWP::UserAgent or Apache::Registry that might be responsible for these problems?

Here's the relevant code: 

#!/usr/bin/perl -w

use strict;
use LWP::UserAgent;
use HTTP::Request::Common qw(POST);
use Time::HiRes qw(gettimeofday tv_interval);

my $ua           = LWP::UserAgent->new();

my $user         = "demo";
my $passwd       = "page";
my $url          = "http://nowhere.com/cgi-bin/bozo.cgi";

my $req          = POST $url,[user=>$user,passwd=>$passwd];
my $start_time   = [gettimeofday];
my $html         = $ua->request($req)->as_string;
my $elapsed      = tv_interval($start_time);

[download]
The server is Solaris 5.7 for Intel, Apache 1.3.12, mod_perl 1.24, perl 5.005_2

Client is Linux-Mandrake 7.1, perl 5.6, LWP::UserAgent 1.69

thanks kindly--

zo.

In reply to slow LWP::UserAgent by johnzo

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.