You can have LWP::UserAgent read a request into a file instead of memory. You can also set it up to use a user defined sub to process the request but it seems it would be trick to try and parse it on the fly (across chunk boundaries.)

From LWP docs:
The request() method can process the content of the response in one of three ways: in core, into a file, or into repeated calls of a subroutine. You choose which one by the kind of value passed as the second argument to request(). The in core variant simply returns the content in a scalar attribute called content() of the response object, and is suitable for small HTML replies that might need further parsing. This variant is used if the second argument is missing (or is undef).

The filename variant requires a scalar containing a filename as the second argument to request(), and is suitable for large WWW objects which need to be written directly to the file, without requiring large amounts of memory. In this case the response object returned from request() will have empty content(). If the request fails, then the content() might not be empty, and the file will be untouched.

The subroutine variant requires a reference to callback routine as the second argument to request() and it can also take an optional chuck size as third argument. This variant can be used to construct "pipe-lined" processing, where processing of received chuncks can begin before the complete data has arrived. The callback function is called with 3 arguments: the data received this time, a reference to the response object and a reference to the protocol object. The response object returned from request() will have empty content(). If the request fails, then the the callback routine will not have been called, and the response->content() might not be empty.

-Lee

"To be civilized is to deny one's nature."