LWP Connection Wasting

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I've got a script that uses LWP::UserAgent to cycle through a given list of URLs and fetch each one. I was noticing that it apparently doesn't take advantage of the HTTP/1.1's ability to limit connection wasting. That is, for each URL, LWP goes through the whole TCP connection setup (3-way handshake) and tears the connection down again after the last of the data for that URL is received. Using HTTP/1.1, you should be able to leave the TCP connection open (e.g. not tear it down) after the first request and send your next request over the same TCP connection. Anybody know if there's a way to make this happen with LWP::UserAgent?

Comment on LWP Connection Wasting

Replies are listed 'Best First'.
Re: LWP Connection Wasting by Corion (Patriarch) on Mar 11, 2004 at 17:11 UTC
In short, no, and it will not happen. In longer, there is LWP::ParallelUserAgent or something like that. To be even more verbose - the problem with connection reuse is, that Perl dosen't know when you're done and there is no way of closing a connection after the timeout has expired unless you do it explicitly. Now that Perl has threads, this could be implemented by spawning a "watcher" thread, but at least under Win32, threads vs. sockets is an interesting topic. Also, the HTTP/1.1 request pipelining means a complete deviation from the LWP request/response model unless you have a good dispatching mechanism and write your programs explicitly to take advantage of it - POE could implement something like it, but POE is ugly, as are all event driven mechanisms.	[reply]
Re: Re: LWP Connection Wasting by bmann (Priest) on Mar 11, 2004 at 20:14 UTC
Hmm. I think you can reuse an existing connection - unless I misunderstand the keep_alive parameter and LWP::ConnCache. These specifically use HTTP 1.1 and reuse existing connections if available, instead of building (and tearing down) a connection for each request.	[reply]