Hi, this is interesting, thanks!

That said let me clarify a bit more on my situation: i prefer to handle the throttle myself. For example, sometimes I will get a server timeout, in which case i will repeat my hit but after sleeping for some longish time (because I know that they are probably doing a backup as it occurs at more or less the same time). Whereas normally I sleep for shorter times in a loop. Some pages I access less often and I would loop over them with a very small sleep value, some other pages I access more frequently and the sleep time must be longer.

Most importantly, I need my sleeps to be variable, seemingly random. Right now, they come out from a random distribution with a mean and a standard deviation which I control.

From the source code of the package you mentioned it looks that it overrides the send_request() method so that it sleeps for a FIXED amount of time and then it does the request. The throttle value (sleep seconds) can be replaced by a throttle function which returns a random number of seconds to sleep drawn from a statistical distribution. That can be useful. However, my need for different throttles on different situations (i.e. GET/POST requests to the same site and not just different websites) still exists.

In reply to Re^2: RFC: LWP::UserAgent hit counter by bliako
in thread RFC: LWP::UserAgent hit counter by bliako

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.