wook has asked for the wisdom of the Perl Monks concerning the following question:

I've written a script in which pages are downloaded via LWP::Simple. I only need the first 300 characters of the file.....so there's no real reason to download the whole thing...it's just slowing the program down. Any suggestions on how I can tell perl to move on once it reaches that 300 character quota?

Replies are listed 'Best First'.
Re: LWP::Simple Acceleration
by duff (Parson) on Oct 03, 2005 at 23:20 UTC

    Use LWP::UserAgent (which is what LWP::Simple is built upon) directly:

    my $ua = LWP::UserAgent->new(max_size => 300); for my $url (@urls) { my $response = $ua->get($url); if ($response->is_success) { print $response->content; } else { die $response->status_line; } }
Re: LWP::Simple Acceleration
by BrowserUk (Patriarch) on Oct 03, 2005 at 23:36 UTC

    300 bytes sounds a little arbitrary? If your need is only to confirm that the url in question is valid and will return content, consider the LWP::Simple head() function.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
    "Science is about questioning the status quo. Questioning authority".
    The "good enough" maybe good enough for the now, and perfection maybe unobtainable, but that should not preclude us from striving for perfection, when time, circumstance or desire allow.
Re: LWP::Simple Acceleration
by Anonymous Monk on Oct 03, 2005 at 23:24 UTC
    If you want to do something that isn't simple, don't use LWP::Simple. Use its bigger brother LWP::UserAgent. When invoking get, set the :content_cb and :read_size_hint fields, and die from the handler you set with :content_db after receiving 300 bytes.

    Alternatively, forget about LWP::*. HTTP is quite simple, after the TCP handshake, it's one message from the client to the server, followed by one message from the server to the client. So, just open a socket, write your request on the socket, read back the reply, close the socket after reading 300 bytes.