Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I have a Perl program that forks multiple times. Each child process needs to append to the same file - potentially at the same time. Seeing as they are just appending to the file is there any need to lock the file or is this just an issue if every child was blowing away the file and re-creating it? Thanks.

Replies are listed 'Best First'.
Re: (OT?) File Locking
by tachyon (Chancellor) on Apr 14, 2002 at 14:58 UTC

    Part of the rational for modern relational database was to solve the problems caused by race conditions and data corruption when using flat file databases that are accessed by mutiple processes simultaneously. Using a database allows the database to handle these issues for you so you might do well to consider that option.

    That said this code should (probably :-) work OK for you:

    #!/usr/bin/perl -w use strict; use POSIX ":sys_wait_h"; use Fcntl ':flock'; $|++; # set a timeout in second waiting for an exclusive lock my $timeout = 10; # in main body of code open file for append # filehandle will be available to all your kids open FH, ">>$file" or die "Can't open $file: $!"; my $pid = fork() die "Can't fork $!\n" unless defined $pid; unless ($pid) { # in a kid, get an exclusive lock on file my $tries = 0; until (flock FH, LOCK_EX|LOCK_NB) { sleep 1; die "Can't get exclusive lock $!\n" if $tries++ > $timeout } seek FH, 0, 2; # make sure we are at EOF print FH $blah; # safe to print here flock FH, LOCK_UN; # release lock for other kids exit; # kill kid } # wait for all kids to finish my $kids; do{ $kids = waitpid(-1, &WNOHANG) }until $kids == -1; close FH; # close file in parent

    Update

    Added seek FH, 0, 2 as per suggestion by dws

    cheers

    tachyon

    s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print

      For this code

      until (flock FH, LOCK_EX) { sleep 1; die "Can't get exclusive lock $!\n" if $tries++ > $timeout }
      either change LOCK_EX to LOCK_EX|LOCK_NB or just drop the whole loop counting retries because flock(FH,LOCK_EX) simply waits until the lock is obtained.

      Also, the seek isn't needed on any platform I know of. Windows needs the locking but already seeks each time you write to a file in append mode (though it certainly won't hurt). And I'd be sure to turn on "autoflush" for the file handle since you are using print to write the data.

              - tye (but my friends call me "Tye")
Re: (OT?) File Locking
by perlplexer (Hermit) on Apr 14, 2002 at 13:47 UTC
    You need to open that file for appending and lock it.

    use Fcntl ':flock'; # ... code ... open FH, ">>$file" or die "Can't open $file: $!"; flock FH, LOCK_EX; # lock it # ... print() to file here flock FH, LOCK_UN; # not really necessary since close() # will unlock the file anyway close FH; # ... code ...


    --perlplexer
      Eeek! No! That code snippet is incomplete, and will eventually get people into trouble if they're trying to run on multiple platforms.

      If you intent to append to a file, you need to handle the small, but very real timing window between opening the file and acquiring the lock. Within that window, a different process can write the the file, leaving you with a bogus seek pointer. Best case, you'll lose the last record(s) written. Worse case, you'll write into the midst of them, leaving corrupted records.

      To mitigate the timing window, add the following line

      use Fcntl ':flock'; ... open FH, ">>$file" or die "Can't open $file: $!"; flock FH, LOCK_EX; # lock it +++ seek FH, 0, 2; # in case eof moved # ... print() to file here flock FH, LOCK_UN; close FH;
        Under UNIX, a file opened for appending automatically seeks to the end on every write(). No locking should be required, even with multiple processes appending, as long as every write is accomplished with a single system call (you can check that with strace/truss). If you take a look at the apache source, it doesn't lock the log file when it writes a message, and that's certainly a case where many processes are appending to the same file at once.

        In perl, you'll want to set $| on the log file handle, or the output buffers from the different processes will interleave. Or use syswrite.

        I don't know if this is true in Windows.

Re: (OT?) File Locking
by {NULE} (Hermit) on Apr 14, 2002 at 19:37 UTC
    Hi AM,

    Different operating systems hande this in different ways. I'm am almost 100% positive that on any UNIX like operating system the append writes are atomic. Meaning as long as you are writting in append mode you don't have to explicitly lock the file. The operating system will make sure that the write from one process will complete before the write from another process begins. From the open() man page (on Mandrake 8.1):

    The file is opened in append mode. Before each write, the file pointer is positioned at the end of the file, as if with lseek. O_APPEND may lead to corrupted files on NFS file systems if more than one process appends data to a file at once. This is because NFS does not support appending to a file, so the client kernel has to simulate it, which can't be done without a race condition
    I take this to mean that with the exception of writes to an NFS mounted file system writes in append mode happen atomically and each write should finish before the system allows another to begin. This means your output may be intermingled with other output, but not corrupted.

    The only link I could find that explicitely states this is here - a manpage for fopen():

    Opening a file with append mode (a as the first character in the mode argument) causes all subsequent writes to the file to be forced to the then current end-of-file, regardless of intervening calls to fseek(3S). If two separate processes open the same file for append, each process may write freely to the file without fear of destroying output being written by the other. The output from the two processes will be intermixed in the file in the order in which it is written.
    Sorry this is vague, but I hope it helps. I know I read this somewhere - I just wish I could recall where for you. Good luck,
    {NULE}
    --
    http://www.nule.org
Re: (OT?) File Locking
by mirod (Canon) on Apr 14, 2002 at 17:54 UTC

    This subject has been dealt with previously (a number of times! ;--) The best answer I've seen, using a lock on a separate file to ensure there is no race condition, is KM's RE: File Locking.