Re^5: perl -Dusethreads compilation

So you don't really want a semaphore (people rarely need that), you just want a Thread::Queue for example, to distribute the jobs among your worker threads. Alternatively, if threads are unavailable to you, see Parallel::ForkManager, as your child/worker threads don't really need to communicate with the master thread.

Also, you might think about whether multiple threads will really improve the runtime of a process that is basically limited by the available bandwith and not by the available computation power.

Comment on Re^5: perl -Dusethreads compilation

Replies are listed 'Best First'.
Re^6: perl -Dusethreads compilation by almut (Canon) on Apr 05, 2010 at 15:52 UTC
you might think about whether multiple threads will really improve the runtime... In case the bandwidth bottleneck in on the servers side, or the servers are responding slowly (think of sites like this one), parallelizing the downloads could certainly help. (update: rephrased to use plural "servers" to make it clearer what I meant)	[reply]
Re^7: perl -Dusethreads compilation by Corion (Patriarch) on Apr 05, 2010 at 16:01 UTC
I'm not sure I understand this. If the server is already pumping out 100% of its upstream bandwidth, how will adding more threads to hammer the server improve throughput? You might get a bigger share of the total bandwidth (and a ban), if the server allocates all connections equally.	[reply]
Re^8: perl -Dusethreads compilation by almut (Canon) on Apr 05, 2010 at 16:09 UTC
...how will adding more threads to hammer the server improve throughput? Sure, if all the downloads are from the same server, hammering likely won't help... The OP's description, however, left it open to interpretation whether all the download requests are actually going to the same server.	[reply]