LWP Byte Limit

vbrtrmn has asked for the wisdom of the Perl Monks concerning the following question:

Hey All, I'm dealing with some large XML files, which I only need the first few lines. I don't really want to get the whole file, is there a nice way, using LWP to only get the first say 5k? I'm using the really basic HTTP::Request...

use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
   $ua->agent("Faux Browser/1.0");
my $req = HTTP::Request->new(GET=>$url);
   $req->content_type('application/x-www-form-urlencoded');
   $req->content('query=libwww-perl&mode=dist');
my $res = $ua->request($req);
if ($res->is_success) {
   # do stuff
} else {
   # error out
}
[download]

--
paul

Comment on LWP Byte Limit Download Code

Replies are listed 'Best First'.
Re: LWP Byte Limit by Aristotle (Chancellor) on Aug 18, 2004 at 18:29 UTC
As I posted in LWP: Downloading First 2KB of an HTML File: `my $data; my $res = $ua->request( $req, sub { $data = shift; die }, 5120 );` [download] Note that `$res->content` will be empty. Makeshifts last the longest.	[reply] [d/l]
•Re: LWP Byte Limit by merlyn (Sage) on Aug 18, 2004 at 18:57 UTC
You can set a `max_size` during the `new` for your useragent. The default value of `undef` sucks down the entire response. -- Randal L. Schwartz, Perl hacker Be sure to read my standard disclaimer if this is a reply.	[reply]
Re: LWP Byte Limit by Fletch (Bishop) on Aug 18, 2004 at 18:30 UTC
Theoretically you should be able to set the `Content-Range` header to the first chunk you want, but keep in mind that servers aren't required to implement handling this header (and may not be able to for things like dynamically generated documents). See section 14.16 of RFC 2616.	[reply] [d/l]