in reply to Looking for a better way to get the number of lines in a file...

I posted Perl code on my blog that solves this problem using windowed sampling and averaging. If anyone arrives at this thread with files that have millions of lines... it's useful. Reposting the code here for anyone to review:
#!/usr/bin/perl use Getopt::Long; $w=200000; # window size $x=10; # segment count GetOptions("only"=>\$only, "window=i"=>\$w, "segs=i"=>\$x); die usage() unless @ARGV; die usage() if $only && @ARGV > 1; $x-=1; for $f (@ARGV) { my $s = -s $f; open (IN, $f) || die "$f: $!\n"; my $rc = 0; my $nl = 0; if ($s > ($w*2)) { for ($i=0;$i<$x;++$i) { seek(IN, $i*($s-$w)/$x, 0) if $i > 0; read(IN, $d, $w/$x); #whole lines only, prevents overcounting if ($i > 0) { $d=substr($d, index($d, "\n")+1); } $d=substr($d, 0, rindex($d, "\n")); $rc += length($d); $nl += xlc($d); } seek(IN, -$w/3, 2); read(IN, $d, $w/3); #whole lines only $d=substr($d, index($d, "\n")+1); $rc += length($d); $nl += xlc($d); } else { $rc += read(IN, $d, $w); $nl += xlc($d); } $nl = int($rc > 0 ? $nl * $s / $rc : 0); $tl += $nl; print "$nl\n" if $only; $nl{$f} = $nl if !$only; } if (!$only) { $m=length($tl)+1; for $f (@ARGV) { printf "%*d %s\n", $m, $nl{$f}, $f; } printf "%*d %s\n", $m, $tl, "total" if @ARGV > 1; } sub xlc { my $d = $_[0]; my $p=0; my $c=0; my $i=0; while (($i=index($d, "\n", $p))>=0) { ++$c; $p=$i+1; } return $p<length($d) ? $c+1 : $c; # correct count } sub usage {<<EOF Usage: alc [-o] <file1> [<file2>] ... Approximate line counts for each file. Attempts to be somewhat compatible with "wc -l" by default. -o|--only Output line count only for a single file. -w|--window <int> Read <int> bytes from head, mid, and tail. -s|--segs <int> Divide file & window into <int> segments. EOF }; __END__ =head1 NAME alc - Approximate line count =head1 DESCRIPTION Approximate line counts for each file. Attempts to be somewhat compatible with "wc -l" by default. =head1 AUTHOR Erik Aronesty C<earonesty@cpan.org> =head1 LICENSE This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself. See L<http://www.perl.com/perl/misc/Artistic.html>. =cut
  • Comment on Re: Looking for a better way to get the number of lines in a file...
  • Download Code

Replies are listed 'Best First'.
Re^2: Looking for a better way to get the number of lines in a file...
by simul (Novice) on Sep 07, 2012 at 16:24 UTC
    Code on the blog has been updated. Always accurate for predictive/repeated files. Does an accurate linear model estimate with gzipped files. Current source: documentroot.com/alc