LWP::Parallel is slow downloader?

dannoura has asked for the wisdom of the Perl Monks concerning the following question:

hi,

I have a script which downloads several hundreds of webpages. Not too long ago I switched from LWP to LWP::Parallel. The problem is that the downloading seems to go much slower than before. I tried use LWP::Debug qw(+); to try to find out what's causing the delays, but couldn't find anything useful. Are there any suggestions about where I could find something to help me figure out why the downloading is so slow?

p.s. I'm sending out 5 parallel requests each time.

Comment on LWP::Parallel is slow downloader? Download Code

Replies are listed 'Best First'.
Re: LWP::Parallel is slow downloader? by perrin (Chancellor) on Apr 13, 2005 at 13:31 UTC
That's right, LWP::Parallel is slow. If you want to go fast, use forking or a different tool altogether. Things like HTTP::GHTTP or HTTP::MHTTP tend to be much faster than LWP.	[reply]
Re: LWP::Parallel is slow downloader? by webchalkboard (Scribe) on Apr 13, 2005 at 14:39 UTC
Maybe you want to play around with the $pua->timeout (2); settings, or as the monk above said, fork off processes manually rather than using the LWP::Parallel, I found it a bit slow when I was using it ages ago as well. Learning without thought is labor lost; thought without learning is perilous. - Confucius WebChalkboard.com \| For the love of art...	[reply]
Re: LWP::Parallel is slow downloader? by ghenry (Vicar) on Apr 13, 2005 at 13:32 UTC
Could we see some code? Thanks. Walking the road to enlightenment... I found a penguin and a camel on the way..... Fancy a yourname@perl.me.uk? Just ask!!!	[reply]