It would be interesting to see how the various options stack up against each other, but they would all ned to be run from the same place to get anything like a reasonable comparison.
From where I am, it makes little or no difference to the total time whether I do all 10 in parallel or serially. The bottleneck is entirely the narrowness of my 40K pipe. YMMV.
To that end, here's a threaded solution:
#! perl -slw use strict; use threads; use Thread::Queue; use LWP::Simple; use Time::HiRes qw[ time ]; $|=1; our $THREADS ||= 3; our $PATH ||= 'tmp/'; sub fetch { my $Q = shift; while( $Q->pending ) { my $url = $Q->dequeue; my $start = time; warn "$url : " . getstore( "http://$url/", "$PATH$url.htm" ) . "\t" .( time() - $start ) . $/; } } my $start = time; my $Q = new Thread::Queue; $Q->enqueue( map{ chomp; $_ } <DATA> ); my @threads = map{ threads->new( \&fetch, $Q ) } 1 .. $THREADS; $_->join for @threads; print 'Done in: ', time - $start, ' seconds.'; __DATA__ www.google.com www.yahoo.com www.amazon.com www.ebay.com www.perlmonks.com news.yahoo.com news.google.com www.msn.com www.slashdot.org www.indymedia.org
In reply to Re: What is the fastest way to download a bunch of web pages?
by BrowserUk
in thread What is the fastest way to download a bunch of web pages?
by tphyahoo
| For: | Use: | ||
| & | & | ||
| < | < | ||
| > | > | ||
| [ | [ | ||
| ] | ] |