The speed with which a remote server responds to an HTTP request is not within the scope of your local Perl program. The only way that your local program could have an effect on how the server responds would be to know if the remote server uses some metric to selectively throttle requests, and if so, to do your best to program around that remote-servers behavior.

In other words, in all probability, the remote server is just slow sometimes. However, it's possible that the remote server is detecting your UserAgent by name (or by IP, or by lack of accepting cookies, or some other metric), and selectively choosing to slow down your requests. Either way, it's not an LWP::Simple issue, or even a Perl issue, the remote server is the one who decides how long requests take, not LWP::Simple. If it does turn out that you're being throttled based on some metric, the solution would be to uncover what criteria are resulting in the server slowing down its responses to your requests, and then try to not fall within those criteria. Maybe you need to accept cookies. Maybe you need to rename your user agent. Maybe you need to originate from different IP's, or to slow down the rate at which you're making requests.

Your best bet, assuming you're living within the TOS of the server you are hitting, is to ask the administrator of that server why his server is slow, and if it's because you're falling into some criteria for the engagement of a safety mechanism, work with the admin to find ways to avoid tripping the behavior.


Dave

In reply to Re^3: use LWP::Simple slows script down. by davido
in thread use LWP::Simple slows script down. by Jesse Smith

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.