Re: How do I retrieve a piece of a file? (use a callback)

You could hang a callback off the request, and die after you've seen 4096 characters. Your script won't actually die, you'll just return back from the request method. Something like this:

use HTTP::Request;
use LWP::UserAgent;

my $html     = '';
my $request  = HTTP::Request->new(GET => "$url/$file" );
my $ua       = LWP::UserAgent->new;
my $response = $ua->request($request, \&cb);

sub cb {
    $html .= $_[0];
    die if length($html) > 4096;
}
[download]

Note that you might want to trim the $html variable back to exactly 4096 chars with substr($html, 0, 4096). Also, you may have asked the question because you know that what you are looking for is somewhere within the first 4k. Of course, if you find what you are looking for earlier, then you can die all that much earlier.

Note that this is about as efficient as it gets; you are getting chunks of the page more or less as they are peeled off the socket and then dealing with them on the fly.

Hmm... I never noticed the $self->{ua} thing before. I'll have to take a closer look at that... or hang on, is that just part of a larger object?

`g r i n d e r`

Comment on Re: How do I retrieve a piece of a file? (use a callback) Select or Download Code

Replies are listed 'Best First'.
(crazyinsomniac) Re^2: How do I retrieve a piece of a file? (use a callback) by crazyinsomniac (Prior) on Nov 24, 2001 at 05:53 UTC
Silly grinder, length is for kids :D (request takes an additional arg, which is, bytesize) `use HTTP::Request; use LWP::UserAgent; my $html = ''; my $request = HTTP::Request->new(GET => "$url/$file" ); my $ua = LWP::UserAgent->new; my $response = $ua->request($request, \&cb, 4096); sub cb { print $_[0]; die; }` [download] ___crazyinsomniac_______________________________________ `Disclaimer: Don't blame. It came from inside the void` `perl -e "$q=$_;map({chr unpack qq;H;,$_}split(q;;,qH*));print;$q/$q;"`	[reply] [d/l]
Re: (crazyinsomniac) Re^2: How do I retrieve a piece of a file? (use a callback) by grinder (Bishop) on Nov 25, 2001 at 01:20 UTC
Yeah, but the person said that trying it that way didn't work as expected. I was giving her/him another way of doing it. Correct me if I'm wrong, but in your example if the remote site is (for instance) heavily loaded and sends you a chunk of (for example) 160 bytes, your callback will be called, die, and close the connection prematurely. later: I suppose what I am talking about is this in another form. -- `g r i n d e r`	[reply]
Re: Re: How do I retrieve a piece of a file? (use a callback) by jmarans (Novice) on Nov 25, 2001 at 07:51 UTC
The $self->{ua} is there because I hacked out _init() from the original .pm, and made it one of my methods so I could watch a bit closer. I like your call back idea, though. It means I don't have to understand why a server I thought is 1.1 compliant isn't. I probably need less than 4k. It was an arbitrary length I chose to transfer; tar will pull out what I need and complain to dev null about the rest. So, this response taught me something useful, and your subsequent posting looks amazing. Thanks.	[reply]