geohar has asked for the wisdom of the Perl Monks concerning the following question:

Hi there, I'm writing an xml backend for a project. I need to be able to lock a file for exclusive access, read/modify it to a second file, then rename the file back over the original. eg. 
use Fctl qw(:flock);

my $fh = new IO::File("file1","r");
flock($fh,LOCK_EX) or die "can't get lock";
my $fh2 = new IO::File("file2","w");
# read fh and write fh2 (ie do a filter)
# move fh2 over fh
# rename("file2","file1"); doesn't work
# File::Copy's move($fh2,$fh1); doens't work
flock($fh,LOCK_UN);

[download]
But.... I can't find a reliable method to rename a file that works without releasing the lock first. The standard rename doesn't work, neither does File::Copy's move. Both barf if the file you want to write over is locked. Is there any way to do this or am I forced to open both files read-write (sysopen style), perform the filter, then seek both to the start and copy in reverse. Is that the only way? Help much appreciated.

... update
I think the crux of it is whether it is always valid to do the following: 
#!/usr/bin/perl

use IO::File;
use Fcntl qw(:DEFAULT :flock);

print "opening file1\n";
  my $fh1 = new IO::File("file1",O_RDONLY | O_CREAT);
print "locking file1\n";
  flock($fh1,LOCK_EX) or die "failed to lock file1";
print "opening file2\n";
  my $fh2 = new IO::File("file2",O_WRONLY | O_CREAT);
print "writing to file2\n";
  print $fh2 "file2";
  close($fh2);
print "renaming file2 over file1\n";
  rename("file2","file1") or die "failed to rename file2 over file1";
print "unlocking file1\n";
 #flock($fh1,LOCK_UN);#probably not needed as close does it
                                 #(might cause issues with buffering?)
 close($fh1);

[download]
or whether the renaming of file2 over file1, whilst we have file1 open is only valid on a unix-style system, or is it guaranteed to work, provided we have a valid flock etc.

Update...
The above code works under OSX, under linux (so I assume it'll work on most unix variants). However, it fails at the rename under Windows. Is that to be expected, or not?

Update...
It appears that renaming an open file is not always allowed - from perlport:
Some platforms can't delete or rename files held open by the system. Remember to close files when you are done with them. Don't unlink or rename an open file. Don't tie or open a file already tied or opened; untie or close it first.

Update...
The above code was neat, but wouldn't work on windows / non-unix? The code below uses a separate lock/semaphore file, so is less neat but works either way. Here it is: 
#!/usr/bin/perl
use IO::File;
use File::Copy;
use Fcntl qw(:DEFAULT :flock);

print "opening and locking file~\n";
  my $fh_lock = new IO::File("file~",O_RDWR | O_CREAT);
  flock($fh_lock,LOCK_EX) or die "failed to lock file1";
print "opening file1\n";
  my $fh1 = new IO::File("file1",O_RDWR | O_CREAT);
print "opening file2\n";
  my $fh2 = new IO::File("file2",O_WRONLY | O_CREAT);
print "reading from file1\n";
  seek($fh1,0,2);
  close($fh1);
print "writing to file2\n";
  print $fh2 "file2";
  close($fh2);
print "renaming file2 over file1\n";
  rename("file2","file1")
    or print "failed to rename file2 over file1, $!\n";
print "unlocking file~\n";
  #flock($fh_lock,LOCK_UN);    #bad with older flock implementations
  close($fh_lock);
  unlink("file~")
    or print "couldn't remove lock - someone else must have it";

[download]

Replies are listed 'Best First'.
•Re: Update in-place with temporary file and flocking
by merlyn (Sage) on Feb 18, 2004 at 16:43 UTC
[reply]
      What's worrying me is that I'll be half way through reading the file whilst it's swapped out from under me. I don't see how that code changes things - but I'm probably missing something.
[reply]
        Once you have the file open, renaming it doesn't matter. The only problem is if someone edits the very file you have open, and the referenced code ensures that this never happens. That's what's nice about an atomic rename operation.

        -- Randal L. Schwartz, Perl hacker
        Be sure to read my standard disclaimer if this is a reply.

[reply]
Re: Update in-place with temporary file and flocking
by borisz (Canon) on Feb 18, 2004 at 16:50 UTC
    What about 
    rename "file1", "file2";
my $fh = new IO::File("file1","w")
flock($fh,LOCK_EX) or die "can't get lock";
my $fh2 = new IO::File("file2","r");
#read fh2 write to $fh
flock($fh,LOCK_UN);
# perhaps delete file2

    [download]
    Boris
[reply]
[d/l]
      Sure, that's what I was going to try next. I was wondering though, If I could get away without the second read-write sequence. I wanted to just rename the 2nd file over the top...
[reply]
        I can not see a second read-write sequence. The operations are the same for both examples.
        Boris
[reply]
Re: Update in-place with temporary file and flocking
by graff (Chancellor) on Feb 19, 2004 at 05:52 UTC
    Whenever I have a situation where multiple concurrent users or processes need brief exclusive access to some shared resource (usually to rewrite or append to a common data file), I tend to use a separate "semaphore" file -- in fact, I've needed this often enough that when I found a TPJ article by Sean Burke discussing semaphores in detail, I put his demo code into a module, which I now use regularly (there's a link to the article in the module docs).

    The basic idea is that the semaphore file is always empty -- it exists purely to establish a lock on some particular resource. Get the lock on the semaphore file, and the resource is yours until you release the semaphore. This eliminates all the loop-holes, race conditions and weirdness of trying to grab and hold a lock on something that you're trying to change.

    Of course, this only works in the cases where everyone agrees to cooperate (or the project is designed to enforce cooperation) and "update access" to the shared resource is always moderated via the specified semaphore -- it won't help in the case where some folks think they can ignore the semaphore stuff and just start hacking away at the actual data file (e.g. with vi or whatever).

[reply]
      Agreed. That'll probably be the solution I have to use. I was hoping however, to be able to get away with locking only the file I intended to update as this seemed neater. Clearly it's not as portable as I'd have thought though. A separate semaphore file would avoid rename failing because the destination file is locked on windows. (Neither the temporary, nor the real file would be locked at all). Just seems a shame that the above code isn't going to work reliably.
[reply]
Re: Update in-place with temporary file and flocking
by cees (Curate) on Feb 18, 2004 at 18:46 UTC

    How about IO::AtomicFile? I don't think it will eliminate the possibility of 2 writers at the same time, but you could combine this module with a lock file to gain exclusive write access to the file (you could block readers this way as well if you needed to).

    - Cees

[reply]
Re: Update in-place with temporary file and flocking
by Anonymous Monk on Feb 18, 2004 at 16:40 UTC
    why can't you modify it in memory and then write it straight to the first file?? just a thought, rune
[reply]
      It's potentially large, and though I could use XML::SAX events to build the whole thing in memory, I'd rather not. Furthermore, only some of the information in the file is of interest/needs altering - the rest needs to just be left alone, so streaming it though as a copy-file-like operation seems like a good idea, rather than using up memory.
[reply]

Back to Seekers of Perl Wisdom