I posted Perl code on my blog that solves this problem using windowed sampling and averaging. If anyone arrives at this thread with files that have millions of lines... it's useful. Reposting the code here for anyone to review:
#!/usr/bin/perl use Getopt::Long; $w=200000; # window size $x=10; # segment count GetOptions("only"=>\$only, "window=i"=>\$w, "segs=i"=>\$x); die usage() unless @ARGV; die usage() if $only && @ARGV > 1; $x-=1; for $f (@ARGV) { my $s = -s $f; open (IN, $f) || die "$f: $!\n"; my $rc = 0; my $nl = 0; if ($s > ($w*2)) { for ($i=0;$i<$x;++$i) { seek(IN, $i*($s-$w)/$x, 0) if $i > 0; read(IN, $d, $w/$x); #whole lines only, prevents overcounting if ($i > 0) { $d=substr($d, index($d, "\n")+1); } $d=substr($d, 0, rindex($d, "\n")); $rc += length($d); $nl += xlc($d); } seek(IN, -$w/3, 2); read(IN, $d, $w/3); #whole lines only $d=substr($d, index($d, "\n")+1); $rc += length($d); $nl += xlc($d); } else { $rc += read(IN, $d, $w); $nl += xlc($d); } $nl = int($rc > 0 ? $nl * $s / $rc : 0); $tl += $nl; print "$nl\n" if $only; $nl{$f} = $nl if !$only; } if (!$only) { $m=length($tl)+1; for $f (@ARGV) { printf "%*d %s\n", $m, $nl{$f}, $f; } printf "%*d %s\n", $m, $tl, "total" if @ARGV > 1; } sub xlc { my $d = $_[0]; my $p=0; my $c=0; my $i=0; while (($i=index($d, "\n", $p))>=0) { ++$c; $p=$i+1; } return $p<length($d) ? $c+1 : $c; # correct count } sub usage {<<EOF Usage: alc [-o] <file1> [<file2>] ... Approximate line counts for each file. Attempts to be somewhat compatible with "wc -l" by default. -o|--only Output line count only for a single file. -w|--window <int> Read <int> bytes from head, mid, and tail. -s|--segs <int> Divide file & window into <int> segments. EOF }; __END__ =head1 NAME alc - Approximate line count =head1 DESCRIPTION Approximate line counts for each file. Attempts to be somewhat compatible with "wc -l" by default. =head1 AUTHOR Erik Aronesty C<earonesty@cpan.org> =head1 LICENSE This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself. See L<http://www.perl.com/perl/misc/Artistic.html>. =cut

In reply to Re: Looking for a better way to get the number of lines in a file... by simul
in thread Looking for a better way to get the number of lines in a file... by coec

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.