in reply to Re^3: Synchronizing multiple processes retrieving files out of shared directory
in thread Synchronizing multiple processes retrieving files out of shared directory

if you were to create sibling or sub-directories on the same disk volume as the one where the depository directory exists, then the perl-internal rename function will still work (and still be atomic in the same way).

Is that true for NFS mounted volumes?

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
  • Comment on Re^4: Synchronizing multiple processes retrieving files out of shared directory

Replies are listed 'Best First'.
Re^5: Synchronizing multiple processes retrieving files out of shared directory
by graff (Chancellor) on Mar 21, 2014 at 04:09 UTC
    I suppose that would depend on what type of OS was being used to export the NFS volume. If it's a POSIX system, then it's using the same sort of OS-level system library call at the server, and given that old and new paths are on the same disk volume (we're just moving an inode out of one directory into another) it will be atomic, so the net result will be the same: no two separate processes will be able to rename the same file.

    That said, I just found a report that some (slightly) older versions of MacOSX were not actually using an atomic implementation of rename - but it seems that's been fixed.

[reply]
[d/l]
Re^5: Synchronizing multiple processes retrieving files out of shared directory
by Ravenhall (Beadle) on Jun 04, 2015 at 15:51 UTC

    I'm working on something very similar, and I can tell you for a fact that NFS complicates things considerably.

    These resources were incredibly helpful:

    Essentially, the first time you ask NFS for a file (via stat or open) it will check the filesystem. For the next 3 seconds (by default) , any stat call is checked against the attribute cache instead of the actual file. This means that even though you check for existence (-e $file), NFS can and will lie to you.

    With a directory, the same thing is true - NFS has a filehandle cache. The first time you readdir on the directory, NFS caches that response for 3 seconds afterwards. So files can be deleted by other NFS clients and (again) NFS will lie to you.

    From the Linux NFS faq http://nfs.sourceforge.net/:

    After a file is deleted on the server, clients don't find out until they try to access the file with a file handle they had cached from a previous LOOKUP. Using rsync or mv to replace a file while it is in use on another client is a common scenario that results in an ESTALE error.

    The NFS Coding how-to has suggestions on how to work around the NFS cache as a programmer. It also has a C utility that you can download and compile on two different servers that share the NFS storage so you can determine which workarounds work and which don't.

    In general, open + close on the file and touching the directory *should* clear the caches on most Linux/Unix operating systems. VMS and Windows are a bit different though, so definitely utilize the resources above before they vanish into the dark recesses of the interwebs.

[reply]

      Doesn't surprise me. I'v seen similar problems with the MS implementation of the CIFS/SMB protocol.

      In theory atomic rename works; and on a good day (probably a Thursday in June), when no one else is looking, it does. But there are no guarantees.

      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
      In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked
[reply]

In Section Seekers of Perl Wisdom