ecuguru has asked for the wisdom of the Perl Monks concerning the following question:

Hi,
I have a perl script that opens a file and adds a data set at the end of the file. 
$file = "/Library/WebServer/Documents/log.txt";
open (DEST, ">>$file") || die &damn();
print DEST $time;
print DEST ",";
print DEST $StationId;
print DEST "\n";
close DEST;

[download]
I'm worried about server load. Does the above code actually load the file into memory?
My file sizes are getting around 500meg to 2gig. Is there a more efficient way to add the data to the logfile then what I'm using?
The above ain't broke, but I'm about to get a large influx of data, and I want to make sure that there isn't a better way to add a line to the end of a data file.
thanks
tim

Replies are listed 'Best First'.
Re: Add a line to the end of a file
by hardburn (Abbot) on Feb 17, 2004 at 22:24 UTC

    Append operations don't read the file into memory. They just set the write pointer to the current end of the file.

    Some OSes and file systems have problems reading files above 2 or 4 GB (usually 2). This is the limit of 32-bit file pointers. To get bigger files, you need a system which supports 64-bit file pointers, which puts the size limit many magnitudes larger than any currently available storage device can hold. Win2k and Linux >=2.4.0 supports this. I can't help you about other systems. You'll also need a version of perl compiled to support 64-bit file pointers, but chances are that if your system supports them already, so is your current perl version.

    ----
    : () { :|:& };:

    Note: All code is untested, unless otherwise stated

[reply]
[d/l]
      Solaris 2.6 and later (SunOS 5.6 -> 5.9) can handle files larger than this size as well. If you are using veritas filesystem you may need to make sure you are mounting with the largefiles option.

      The only other problem you will want to avoid here is to make sure that you don't have any race conditions if you run the append on multiple threads or processes at the same time. You will want to look into locking in that case, or maybe a service that handles writing to the file and takes the input from a port or queue (think like syslog or sendmail).


      -Waswas
[reply]
        If you open a file with O_APPEND (which is what >> does) then each write to the file is preceded with a seek to the current end of the file. The seek and write are atomic, provided that you are writing less than a block of data (might work for more than a block, I disremember). The size of a block varies from one filesystem to another and depends on the platform, the size of the filesystem, the options with which the fs was created, the phase of the moon, and so on. It is *typically* of the order of 1 to 4 K.

        Be aware that this race condition *does* exist in NFS. Which doesn't provide adequate locking to get around it. Yay.

[reply]
[d/l]
[select]
Re: Add a line to the end of a file
by Abigail-II (Bishop) on Feb 17, 2004 at 22:48 UTC
    Does the above code actually load the file into memory?
    Only if your OS requires to do so, but I don't know any OS that requires it.
    Is there a more efficient way to add the data to the logfile then what I'm using?
    It depends on how often this code is run. If you log a lot of lines this way, opening and closing the file for each piece of data being logged, it's inefficient. It would be better to open the file once, and close if the program terminates.

    Abigail

[reply]
Re: Add a line to the end of a file
by davido (Cardinal) on Feb 18, 2004 at 06:59 UTC
    In the spirit of "print seldom and print late", here's a little cleaner and maybe more efficient method: 

    my $file = '/Library/WebServer/Documents/log.txt';
open ( DEST, '>>', $file ) or die "Bleah!\n$!";
print DEST "$time, $StationId\n";
close DEST;

    [download]


    Dave

[reply]
[d/l]
Re: Add a line to the end of a file
by Skeeve (Parson) on Feb 17, 2004 at 22:18 UTC
    It shouldn't load the file into memory. There is no reason for it.

    I also don't think that there is a better way to do it.

    But this ain't a perl question,
[reply]

      I also don't think that there is a better way to do it.

      How about just: 

      echo 'stuff' >> file.txt

      [download]

      This will be much faster and use buckets less memory than firing up a full blown perl interpretter. Works on Win32 or *nix. It's shorter. Better? Depends what you need.... As always YMMV. 

      C:\>echo Hello World! > better

C:\>date /T >> better

C:\>type better
Hello World!
Tue 17/02/2004

C:\>

      [download]

      cheers

      tachyon

[reply]
[d/l]
[select]
        Did he say the appending to the log is the only thing his script does? Nope, he didn't. So as usualy I'd assume that this logging is just a part of a more complex perl script. An echo wouldn't be of much use, don't you aggree?

        Update (at -1): Yes! Come on! Downvote me! Hey! What's wrong with this! Please explain...
[reply]

      Actually, it is a perl question, at least in part, though it's not really a Perl question. (See my other reply).

      ----
      : () { :|:& };:

      Note: All code is untested, unless otherwise stated

[reply]
[d/l]

Back to Seekers of Perl Wisdom