in reply to Biting off more with LWP and problems with blocking forks()?

It's very, very slow when fetching a page to take such a small 'nibble'. Is this server-side dependant?

It actually depends on every network and router your traffic has to pass through. Taking small nibbles is built into the nature of the Internet. Normally, TCP tries to keep several packets "in the air" at once, so that the full capacity of the link is always being used. Some sites (notably Yahoo) seem to disable this, though, so you have to wait two complete round-trip times for each packet. There's nothing you can do about that.

BTW, what kind of link are you trying to download all this stuff through? Maybe it's getting congested -- have you tried using fewer processes?

  • Comment on Re: Biting off more with LWP and problems with blocking forks()?