in reply to How can I read just the end of a huge file?

Count the lines in the file ( how to do this is documented in perlfaq ), subtract 20000 from that number and put the result in $start_at, then open the logfile for reading, incrementing the counter for each line read in and ignore until you've hit $start_at ...

Or ... 75MB is not *that* huge these days. You could load the whole file into memory and just grab the last 20000 lines, e.g. 

# get_last returns an array reference for efficiency,
# so we need to de-reference; hence the @{} around
# the subroutine call 
my @lines = @{ get_last("foo.log", 20000) };

sub get_last {
 my ($filename, $from_last) = @_;
 open FILE, $filename or die "Can't open $filename: $!\n";
 my @lines = <FILE>; #loads the whole thing into memory
 close FILE;
 
 # return the last $from_last lines in an anonymous array
 # $#array_name is the index of the last element of 
 # the array, so @lines[$#lines-20, $#lines] is the
 # last 20 elements of @lines
 [ @lines[$#lines-$from_last, $#lines] ];
}

[download]

HTH

Philosophy can be made out of anything. Or less -- Jerry A. Fodor

Replies are listed 'Best First'.
Re: Re: How can I read just the end of a huge file?
by turnstep (Parson) on Mar 12, 2001 at 22:20 UTC
    Count the lines in the file ( how to do this is documented in perlfaq ), subtract 20000 from that number and put the result in $start_at, then open the logfile for reading, incrementing the counter for each line read in and ignore until you've hit $start_at ...

    A far better way is to use seek to jump right to the end of the file. Just go back far enough so that you know you are getting the chunk that you need, since seek deals in bytes, not lines. A quick example: 

    #!/usr/bin/perl

use strict;

my $bigfile = shift or die "Usage: $0 filename <lines>\n";

my $charsperline = 80; ## A rough guess, adjust as needed

my $linesback = shift || 10; ## How many "lines" back to read

open(BIG, $bigfile) or die "Could not open $bigfile: $!\n";

## Make a reasonable guess as to how far back to go.
my $gobackchars = $linesback*$charsperline;

## Figure out how big it is, so we don't go *too* dar back:
my $filesize = -s $bigfile;
$gobackchars = $filesize if $gobackchars > $filesize;

## Go to the end of the file minus that many characters
seek(BIG,-$gobackchars,2);

## We probably arrived in the middle of a line,
## so throw out everything until the start of a new line
## (As determined by $/)
<BIG>;

## Now we print out the rest of the lines:
print while(<BIG>);

close(BIG);

    [download]
    <!- Bam! -->

    Update: Sorry, arturo. Some days (especially rainy Mondays) I need large <HUMOR> tags wrapped around things. :)

[reply]
[d/l]
[reply]
        if you're looking for fast code:

        $result=`tail your_file`;
        i do like this method even if it's more scripting than coding...
[reply]

In Section Seekers of Perl Wisdom