in reply to Intelligent logging

File I/O is (almost) completely dependent on the speed of your hard-drives. A 7200rpm disk is going to be half as fast as a 15000rpm disk. Now, depending on the setup of your disk(s), that can be altered. For example, if you use certain RAID operations, that can significantly speed up your reads, but slow down your writes. Additionally, depending on which filesystem you use, that can have an impact. For example, ReiserFS is much better at many small files than ext2 or ext3. Journalling filesystems are generally slow with writes than non-journalling filesystems, for example ext2 is faster than ext3. (If you need journalling, you're not going to comlpain.)

Overall, however, opening and closing files is generally O(1), which means they generally are rather fast. Now, reading and writing data can be more costly. But, the cost is usually in the memory structures people read into than in the actual cost of reading. Writing, especially appending as in a logfile is almost O(0); in other words, it's so negligible as to be irrelevant.

Now, my question is why are you closing the file after every iteration? It sounds like you really only want to flush the buffers after every iteration. So, you could do something like:

use IO::File; my $fh = IO::File->new( ">>$filename" ) or die "Cannot open '$filename +' for appending: $!\n"; # Do your stuff here. At the end of every iteration, call $fh->flush; $fh->close;

Now, an alternative would be to turn autoflushing on, using $|++; at the top of your script, preferably in a BEGIN block. There are many developers, including merlyn and Abigail-II that do that reflexively. I didn't suggest that at first for a few reasons:

  1. That turns autoflushing on for EVERY filehandle. You might not want that.
  2. Buffering is the default for a reason - the act of writing generally costs as much for 30bytes as for 30K. So, you might as well write as few times as possible.
  3. This turns off buffering for reads as well. You might not want that.
  4. It sounds like you want to control when you flush your buffers. Autoflushing takes that control away from you.

You just have to figure out what's best for your specific need.

Being right, does not endow the right to be rude; politeness costs nothing.
Being unknowing, is not the same as being stupid.
Expressing a contrary opinion, whether to the individual or the group, is more often a sign of deeper thought than of cantankerous belligerence.
Do not mistake your goals as the only goals; your opinion as the only opinion; your confidence as correctness. Saying you know better is not the same as explaining you know better.

Replies are listed 'Best First'.
Re^2: Intelligent logging
by chromatic (Archbishop) on Oct 29, 2004 at 01:37 UTC

    I agree that disabling buffering reflexively is silly, but I disagree with points 1 and 3. According to perlvar:

    If set to nonzero, forces a flush right away and after every write or print on the currently selected output channel.

    ... and ...

    This has no effect on input buffering.
Re^2: Intelligent logging
by thor (Priest) on Jul 14, 2005 at 15:27 UTC
    generally O(1), which means they generally are rather fast
    O(1) != fast. Roughly speaking big-o notation says how well a given operation scales based on the size of the input. So, it speaks to the realative speed of say an input of size 1000 vs input of size 10000. That having been said, I'd say that opening a file is not O(1), but probably O(n); opening n files takes n times as long as opening one file. Granted, each one is really fast, but it takes a finite amount of time. There can be operations that are O(1) that are slow. But such operations will be equally slow regardless of whether they're operating on one million. I'll end my pedantary now.

    thor

    Feel the white light, the light within
    Be your own disciple, fan the sparks of will
    For all of us waiting, your kingdom will come