Efficiently send GET requests under a second

jcr has asked for the wisdom of the Perl Monks concerning the following question:

I have a working piece of code which performs a series of HTTP GET requests within a second. I am using LWP::Simple module to perform the GET requests and the Time::HiRes to pause for a fraction of a second between requests. The baseline time it takes to perform a GET request to a certain server is 0.06 sec (or 60,000 msec), so i deduced that in order to perform 10 requests a second i must sleep for 0.04 seconds (Actually, to account for any errors I rounded down the sleeping time to 0.03 sec).

The reason I am posting this is to ask if anyone knows of a better or more efficient way to produce a script that "pounds" a server at 'x' times per second.

Thank You very much,

jcr

here is the code:

use LWP::Simple;
use Time::HiRes qw(usleep);

open(URL_LIST, 'url_list.txt') || die "Cannot open: 'url_list'\n$!\n";
while(<URL_LIST>) {

  chomp;

  get($_);

  usleep('0.03');

}

close(URL_LIST);
[download]

Comment on Efficiently send GET requests under a second Download Code

Replies are listed 'Best First'.
Re (tilly) 1: Efficiently send GET requests under a second by tilly (Archbishop) on Oct 03, 2000 at 15:14 UTC
Don't guess how long you can sleep. If you are using Time::HiRes already, check what time it is, compare to the last time, and use that information to sleep the amount you want.	[reply]
RE: Re (tilly) 1: Efficiently send GET requests under a second by Fastolfe (Vicar) on Oct 03, 2000 at 18:33 UTC
Agreed. A good algorithm might be to divide up time into discrete seconds (Time::HiRes can override time to provide you with hi-resolution timekeeping), like `$time_left = $start_time + 1 - time()`, calculate how many more requests you have to do in the remaining fraction of a second, and divide the time into slices, sleeping an appropriate amount of time so that the next slice occurs on time. That would allow your script to adapt to requests that vary in duration (due to network conditions or loading, etc.).	[reply] [d/l]
Re: Efficiently send GET requests under a second by Jouke (Curate) on Oct 03, 2000 at 11:23 UTC
hmm..my answer disappeared...I'll try again: I don't think there is a secure way to make this run x times a second/minute/whatever, because you can never tell if the GET action always takes these 60msec. It's very well possible due to network traffic, server load or whatever that a certain GET takes longer. F.i. 1 second. So what then?? Jouke Visser, Perl 'Adept'	[reply]