in reply to Re^2: Speeding things up -- LWP::Parallel
in thread Speeding things up

Although 100/sec sounds fun, I rather think it would look like some sort of feeble DDoS attack and get my IP blocked. I've read 5/sec is considered high by some spider writers. Apparently you can register your site with Google and set a parameter to limit it's strike rate though sometimes the Google spider just ignores it.

I don't run a web server, but I bet the logs are just stuffed full of bots gathering pages.

I do wonder what a polite rate is though; fast enough so that old results are still timely but slow enough to not be annoying.

  • Comment on Re^3: Speeding things up -- LWP::Parallel

Replies are listed 'Best First'.
Re^4: Speeding things up -- LWP::Parallel
by hippo (Archbishop) on Jun 17, 2018 at 09:35 UTC
    I don't run a web server, but I bet the logs are just stuffed full of bots gathering pages.

    I do and they are.

    Some web hosters don't care so YMMV. At $WORK we take a more heavy-handed approach. Large sources of traffic get noticed and investigated. Sources of bad traffic (any size) get investigated. If the results of the investigations warrant it, the bad actors get banned from the netork in its entirety.

    As for a polite rate, stick to 1 request per second at most and you won't even show up on anyone's radar. If you absolutely need more than that contact the site in advance as they probably have an API for whatever purpose it is you have. Do as you would be done by.

    Oh, and a single-IP DDoS is called a DoS. ;-)

[reply]

In Section Seekers of Perl Wisdom