Peetie has asked for the wisdom of the Perl Monks concerning the following question:

Im trying to write a script that grabs the last x number of lines of a file on a remote web site and this is what i have so far. These are punkbuster screenshots from a websserver for the americas army game and the html page is quite large. As my code stands now I have to download the entire code from the site to display the last 20 lines. Is there a way I can just download the last 20 lines from the site without having to download the whole page using perl to make for faster download time? 
#!/usr/bin/perl -w 
use IO::Socket; 
$sock = new IO::Socket::INET (PeerAddr => "xx.xx.xx.xx", 
PeerPort => "http(80)"); 

die "Couldnt create socket: $@" unless $sock; 

$sock->autoflush(1); 

$sock->print("GET /xx.xx.xx.xx/pbsvss.htm HTTP/1.1\n Host:xx.xx.xx.xx:
+80\n\n"); 
@document = $sock->getlines(); 
@lastlines = @document[-20 .. -1]; 
print "last lines: \n\n"; 
print @lastlines;

[download]

Replies are listed 'Best First'.
Re: retrieve html code in reverse from remote server
by fizbin (Chaplain) on Oct 20, 2005 at 03:14 UTC
    Is there a way I can just download the last 20 lines from the site without having to download the whole page using perl to make for faster download time?

    Not directly, but there is a way to download only the last certain number of bytes, assuming that the server on the other end supports it. Note that if the server on the other end doesn't support it, you'll just end up downloading the whole thing anyway. For example, this gets the last 500 bytes of a page:

    use LWP::UserAgent;

my $ua = LWP::UserAgent->new;
$ua->timeout(10);
$ua->env_proxy;

my $url = 'http://cpan.uwinnipeg.ca/htdocs/libwww-perl/LWP/UserAgent.h
+tml';

my $response = $ua->get($url, Range => 'bytes=-500');

print $response->content;

    [download]

    So what I would recommend is that you get an estimate of the number of bytes in the last 20 lines, double it, and then do something like: (assuming that the doubled estimate was, say, 3500)

    use LWP::UserAgent;
use strict;

my $ua = LWP::UserAgent->new;
$ua->timeout(10);
$ua->env_proxy;

my $url = 'http://xx.xx.xx.xx/xx.xx.xx.xx/pbsvss.htm';

my $response = $ua->get($url, Range => 'bytes=-3500');

my @lines = split(/\n/, $response->content);
my @lastlines = @lines[-20 .. -1];
print "last lines: \n\n"; 
print @lastlines;

    [download]
    --
    @/=map{[/./g]}qw/.h_nJ Xapou cets krht ele_ r_ra/;
map{y/X_/\n /;print}map{pop@$_}@/for@/

    [download]
[reply]
[d/l]
[select]
Re: retrieve html code in reverse from remote server
by ikegami (Patriarch) on Oct 20, 2005 at 02:24 UTC

    HTTP/1.1 (and maybe earlier) provide a means of requesting byte ranges. You could do something like File::ReadBackwards but that would require multiple fetches. It's probably faster to just download the whole thing.

    Update: Linked fixed.

[reply]

Back to Seekers of Perl Wisdom