in reply to Reporting "percent done" while processing a large file

I am a huge proponent of sttaus messages, but with text files, to do a percent you would have to first get the number of lines (which on large data is a waste). I tend to do the following for text data :
while (my $line = <INFILE>) { if ( ($. % 5000) == 0) { print "Processed $. records\n"; # or, just to let them know # print "."; } ## process away }
Now, being Mr. Binary-data-can-be-done-with-perl, i tend to do alot of binary op's on files, and for that you can do (this is untested, but mostly just to show the idea):
my $TOTAL = (-s $FILE); while (read(INFILE,$buf,$bufsize)) { ## do work $BYTES_READ+=$bufsize; if ( ($BYTES_READ % 5000) == 0) { printf("%.3f %% (%d of %d)\n",($BYTES_READ/$TOTAL)*100,$BYTES_ +READ,$TOTAL); } }

my own worst enemy
-- MZSanford

Replies are listed 'Best First'.
Re: Re: status
by Masem (Monsignor) on Sep 18, 2001 at 17:00 UTC
    Well, with text, you can still use the latter solution as to not determine the number of lines beforehand if the file is not a fixed record file:
    my $total = ( -s $FILE ); my $cummulative = 0; while ( my $line = <FILE> ) { $cummulative += length( $line ); do_processing( $line ); printf( "%2d%% Done\n", $cummulative/$total*100.0 ); }

    -----------------------------------------------------
    Dr. Michael K. Neylon - mneylon-pm@masemware.com || "You've left the lens cap of your mind on again, Pinky" - The Brain
    It's not what you know, but knowing how to find it if you don't know that's important

Re: Re: status
by blakem (Monsignor) on Sep 18, 2001 at 21:31 UTC
    Perhaps its just me, but I always find the construct:
    if ( ( whatever ) == 0 ) { }
    easier to read, when written like so:
    unless ( whatever ) { }
    So, I would have written the above as:
    unless ($. % 5000) { print "Processed $. records\n"; # or, just to let them know # print "."; }

    -Blake
    off to try that cool -s trick for status bars....