You could hang a callback off the request, and die after you've seen 4096 characters. Your script won't actually die, you'll just return back from the request method. Something like this:

use HTTP::Request;
use LWP::UserAgent;

my $html     = '';
my $request  = HTTP::Request->new(GET => "$url/$file" );
my $ua       = LWP::UserAgent->new;
my $response = $ua->request($request, \&cb);

sub cb {
    $html .= $_[0];
    die if length($html) > 4096;
}

[download]

Note that you might want to trim the $html variable back to exactly 4096 chars with substr($html, 0, 4096). Also, you may have asked the question because you know that what you are looking for is somewhere within the first 4k. Of course, if you find what you are looking for earlier, then you can die all that much earlier.

Note that this is about as efficient as it gets; you are getting chunks of the page more or less as they are peeled off the socket and then dealing with them on the fly.

Hmm... I never noticed the $self->{ua} thing before. I'll have to take a closer look at that... or hang on, is that just part of a larger object?

--
g r i n d e r

In reply to Re: How do I retrieve a piece of a file? (use a callback) by grinder
in thread How do I retrieve a piece of a file? by jmarans

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.