acidmax has asked for the wisdom of the Perl Monks concerning the following question:

Fellow monks,
I am processing some large file and would like to have a simple statistic about lines processed per second.

Example output:
Processing stated at $start_time\n Processed $lines lines ($speed lines/second)\r <- this one!

What should I use? I was thinking about emiting SIGALARM every second and counting the difference between current and last interrupt line counters.
Any suggestions would be welcome.

Replies are listed 'Best First'.
Re: Live operations-per-second counter
by Tanktalus (Canon) on Mar 20, 2005 at 20:06 UTC

    Most "speed" counters usually show the overall speed, not the current speed. Thus, you can even just print this out every x number of lines, and just divide the number of lines processed so far by the time elapsed (you may want to use Time::HiRes for that).

    Or, rather than simply every x lines (where x may need to be large to keep from printing to STDOUT too often - STDOUT is actually very slow, so you don't want to waste too much time printing to the console), every x lines (where x can be much smaller) check the time (not HiRes) to see if the seconds have changed, and then print out (using HiRes, or not) the speed.

Re: Live operations-per-second counter
by kvale (Monsignor) on Mar 20, 2005 at 20:01 UTC
    Try Benchmark:
    use Benchmark; $t0 = new Benchmark; # ... your code here ... $t1 = new Benchmark; $td = timediff($t1, $t0); print "the code took:",timestr($td),"\n";

    -Mark

Re: Live operations-per-second counter
by Joost (Canon) on Mar 20, 2005 at 20:37 UTC
      Code runs much slower with profiling turned on, so this method cannot be used to measure times (to say nothing of "in production").
      --kap
Re: Live operations-per-second counter
by acidmax (Sexton) on Mar 20, 2005 at 22:26 UTC
    This is the simple solution I've implemented, following Tanktalus's advice:
    my $stop; my $speed = 0; my $start = time; while (<CONNS>) { $counter++; if ($counter =~ /00$/) { $stop = time; my $diff = $stop - $start; my $tdiff = ($diff == 0) ? 1 : $diff; $speed = sprintf("%.f", ($counter/$tdiff)); } ... line processing ... print "Processed: $counter (at $speed lines/s) \r"; }
    Not so elegant or fast, but works. Thanks everyone!
      if ($counter =~ /00$/) {
      can be written as
      if ($counter % 100 == 0) {
      which saves perl from converting the number to a string and checking it against a regexp.
        if ($counter % 100 == 0) {
        Or even
        unless ($counter % 100) {
        (But then it depends on how much one is familiar with such a construct - for me it is quite as intuitive as the former, but it's obvious that others' mileage may vary)
Re: Live operations-per-second counter
by bpphillips (Friar) on Mar 21, 2005 at 04:32 UTC
    this might not be exactly what you want, but if the goal is to evaluate the progress of the operation, you could maybe look at Term::ProgressBar. I've not personally used it but have kind of kept in mind for the future. Basically, you set up the object with the total number of items to process and then update it throughout the processing and it will show you how much has been completed as well as an estimate of when it will be completed.

    Here's an example straight out of the perldocs for those who don't want to click through to the above:
    #!/usr/bin/perl use Term::ProgressBar 2.00; use constant MAX => 100_000; my $progress = Term::ProgressBar->new(MAX); for (0..MAX) { my $is_power = 0; for(my $i = 0; 2**$i <= $_; $i++) { $is_power = 1 if 2**$i == $_; } if ( $is_power ) { $progress->update($_); } }
    Anyway, maybe not exactly the solution you were looking for but it's something I've been meaning to use and thought it might come in handy for your case.
      Unfortunately, I don't know how many lines does this file contain. I have several other programs, where I could use it. Thanks a lot!
Re: Live operations-per-second counter
by blueberryCoffee (Scribe) on Mar 21, 2005 at 00:15 UTC
    just count and average as you go.
    $setTimeStart = time; $setCount = 0; while (<IN>) { # process line here $setCount++; # show line speed for every 1000 lines so prints don't # slow you down so much if($setCount < 999) { $setCount = 0; $setTime = time - $setTimeStart; print "processing 1000 lines at " . $setTime/1000 . " per line\n +"; $setTimeStart = time; } }
Re: Live operations-per-second counter
by gam3 (Curate) on Mar 21, 2005 at 18:07 UTC
    I like the alarm idea. But using threads would be even better.
    use strict; use Time::HiRes qw ( alarm time usleep sleep); our $starttime; our $lasttime; our $linecount = 0; our $lastlinecount = 0; $SIG{ALRM} = sub { my $now = time; printf "\rline %d %8.1f avg lines per second %8.1f lines per s +econd", $linecount, $linecount/($now - $starttime), ($linecount - $lastlinecount) / ($now - $lasttime); $lasttime = time(); $lastlinecount = $linecount; }; alarm(1, .5); $| = 1; $lasttime = $starttime = time(); while (1) { $linecount++; last if $linecount > 9999; usleep(0.001); }; print "\n"; alarm(0); $SIG{ALRM} = undef;
    -- gam3
    A picture is worth a thousand words, but takes 200K.
      And here is that threaded solution.
      use strict; use Config; $Config{useithreads} or die "Recompile Perl with threads to run this p +rogram."; use threads; use threads::shared; use Time::HiRes qw ( time sleep); our $starttime : shared; our $linecount : shared = 0; our $running : shared = 1; sub progress { my $lasttime = $starttime; my $lastlinecount = 0; sleep(1); $| = 1; do { my $now = time(); printf "\rline %d %8.1f avg lines per second %8.1f lines per s +econd", $linecount, $linecount/($now - $starttime), ($linecount - $la +stlinecount) / ($now - $lasttime); $lasttime = time(); $lastlinecount = $linecount; sleep(.5); } while ($running); }; $starttime = time(); our $thr1 = threads->new(\&progress); while (1) { $linecount++; last if $linecount > 9999; sleep(0.001); }; print "\n"; $running = 0; $thr1->join();
      -- gam3
      A picture is worth a thousand words, but takes 200K.