in reply to managing many parallel HTTP requests

There are some ways to keep both, the comfort of LWP and the parallelism.

One would be to fork() instead of using Perl threads. This method comes with the price of additional IPC, which I'd handle via a database.

Another way would be to use Coro, which will come at the cost of a new XS module to install and some slight instability introduced by Coro, depending on what your platform is.

Yet another way could be to simply spread out the work across more machines of course.

  • Comment on Re: managing many parallel HTTP requests

Replies are listed 'Best First'.
Re^2: managing many parallel HTTP requests
by sgifford (Prior) on Sep 09, 2005 at 15:03 UTC
    If you decide to use fork, take a look at Parallel::ForkManager, which will do much of the dirty work for you.