in reply to Re: Synchronizing multiple processes retrieving files out of shared directory
in thread Synchronizing multiple processes retrieving files out of shared directory

My original goal was to avoid having filename constraints on the content being dropped off into the shared directory. (Original plan included having multiple customer departments dropping off work, but it's turning out that working entering this directory is all coming from the same location, so we have more control over naming than originally expected.)

I'm going to start by trying lock files with exclusive locks on them. If a process can't create a new file with an exclusive lock, then they move on to the next file in the directory. If that doesn't work, I'm going to see if I can implement a rename-based process.

Thanks.

  • Comment on Re^2: Synchronizing multiple processes retrieving files out of shared directory

Replies are listed 'Best First'.
Re^3: Synchronizing multiple processes retrieving files out of shared directory
by graff (Chancellor) on Mar 21, 2014 at 03:07 UTC
    I think if you were to create sibling or sub-directories on the same disk volume as the one where the depository directory exists, then the perl-internal rename function will still work (and still be atomic in the same way). Each process could create its own sub- or sister directory, and the same logic should apply.
[reply]
[d/l]
      if you were to create sibling or sub-directories on the same disk volume as the one where the depository directory exists, then the perl-internal rename function will still work (and still be atomic in the same way).

      Is that true for NFS mounted volumes?

      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
        I suppose that would depend on what type of OS was being used to export the NFS volume. If it's a POSIX system, then it's using the same sort of OS-level system library call at the server, and given that old and new paths are on the same disk volume (we're just moving an inode out of one directory into another) it will be atomic, so the net result will be the same: no two separate processes will be able to rename the same file.

        That said, I just found a report that some (slightly) older versions of MacOSX were not actually using an atomic implementation of rename - but it seems that's been fixed.

[reply]
[d/l]

        I'm working on something very similar, and I can tell you for a fact that NFS complicates things considerably.

        These resources were incredibly helpful:

        Essentially, the first time you ask NFS for a file (via stat or open) it will check the filesystem. For the next 3 seconds (by default) , any stat call is checked against the attribute cache instead of the actual file. This means that even though you check for existence (-e $file), NFS can and will lie to you.

        With a directory, the same thing is true - NFS has a filehandle cache. The first time you readdir on the directory, NFS caches that response for 3 seconds afterwards. So files can be deleted by other NFS clients and (again) NFS will lie to you.

        From the Linux NFS faq http://nfs.sourceforge.net/:

        After a file is deleted on the server, clients don't find out until they try to access the file with a file handle they had cached from a previous LOOKUP. Using rsync or mv to replace a file while it is in use on another client is a common scenario that results in an ESTALE error.

        The NFS Coding how-to has suggestions on how to work around the NFS cache as a programmer. It also has a C utility that you can download and compile on two different servers that share the NFS storage so you can determine which workarounds work and which don't.

        In general, open + close on the file and touching the directory *should* clear the caches on most Linux/Unix operating systems. VMS and Windows are a bit different though, so definitely utilize the resources above before they vanish into the dark recesses of the interwebs.

[reply]

In Section Seekers of Perl Wisdom