I don't run a web server, but I bet the logs are just stuffed full of bots gathering pages.

I do and they are.

Some web hosters don't care so YMMV. At $WORK we take a more heavy-handed approach. Large sources of traffic get noticed and investigated. Sources of bad traffic (any size) get investigated. If the results of the investigations warrant it, the bad actors get banned from the netork in its entirety.

As for a polite rate, stick to 1 request per second at most and you won't even show up on anyone's radar. If you absolutely need more than that contact the site in advance as they probably have an API for whatever purpose it is you have. Do as you would be done by.

Oh, and a single-IP DDoS is called a DoS. ;-)

In reply to Re^4: Speeding things up -- LWP::Parallel by hippo
in thread Speeding things up by AlwaysSurprised

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.