Well , in the code posted, without $|=1 the user won't see any progress message until DONE

perl -e " $|=0; for (1,2,3) { print qq[\r hi n $_] ; sleep 1; } print 
+qq[\r done now \n]; "

perl -e " $|=1; for (1,2,3) { print qq[\r hi n $_] ; sleep 1; } print 
+qq[\r done now \n]; "
[download]

On the other hand, buffering is probably irrelevant if you seek , whether seek flushes the buffers (what probably happens) or discards them

Yes, either a seek() or a tell() will cause the buffers to be flushed to disk even if the filehandle is "non buffered" - the default.

This is the reason why when when you are using a file in read/write mode, you have to have an intervening seek() or tell() when switching between reading or writing. Otherwise the file pointer will get (or can get) confused!

STDERR is not buffered by default, whereas STDOUT is buffered (as are all disk file handles unless you say otherwise).

It should be noted that turning off buffering (or flushing the buffers often) to a disk file will seriously impact performance. In the best case, the disk has to rotate all the way around to write that sector again. In a more typical case, especially on a multi-user system, other disk operations will get sequenced inbetween your writes - causing further slow downs. This doesn't take much CPU time, but the program execution time can increase quite a bit.

If the idea of logging progress to the disk is to find out where the program "crashes" (the OS will flush all data to disk during process cleanup), then use a tell() to find out the byte position of the beginning of the first "progress message", when you are "done", seek back to that spot and write the "done message" rather than doing that on a message by message basis. This will be much faster than overwriting each line as they come in due to reasons above.

Other options: Designing an easily parseable log file format is an art. Sometimes I just put a tag like "STATUS:" at the beginning of the log line for some intermediate status that later will be of no significance if the program's function succeeds. Then when processing the log file, just remove all lines starting with "STATUS:".

A typical disk write buffer will be 2-4K. Writing 100 short lines is no significant performance impact (usually). Flushing the buffers and forcing a disk write for each line will be measurable.

I don't recommend seeking around within a log file because: this file is presumably there to record what got done or what went wrong and opening such a file in read/write mode adds another layer of complexity and a way for the program to "go wrong" (see above for need for intervening seek()/tell() operations).

Update: SuicideJunkie's suggestion of not writing this "progress...5%" stuff to the log file is probably the best idea. If I could give that post 2 votes, I would!

On another point: On some O/S's like Windows, writing to the screen is actually very slow (even the text mode command window! - not just the GUI which is *really* slow). Sometime you can spend more time spewing stuff to the screen than it takes to do it! If this is a command line program and you want the user to believe that it is really running, consider just putting a "." after every 289 records processed or whatever. As long as something happens every 20 seconds or so, the user won't CTL-C if he has been warned that this program will take awhile. Also, a bunch of "."'s don't take up any significant space in the log file and are easy to suppress when you read it back for processing.

For a scheme that is mostly independent of the time-per-iteration, and optimized for users:

if ($lastPrintTime != time)
{
  $lastPrintTime = time;
  printf("Reticulating Splines: (%3d%%)\r", 100 * $loopIndex / $totalW
+ork);
}
[download]

That gets you a print only once per second (sometimes two seconds). Plenty often for humans, but almost never to the computer.

I'm usually lazier than that, and just slap on a cheap modulo conditional after seeing how long the loop iterations take:

printf("Evaluating Widgets: (%3d%%)\r", 100 * $loopIndex / $totalWork) unless ($loopIndex % 200);