in reply to Synchronizing multiple processes retrieving files out of shared directory

Renaming a file within a directory should be a one-step (atomic) process (rename invokes a system an OS-level system library call - but as such, there might be site-specific issues you need to watch out for).

This approach relies on having some reliable constraint on the names of files being dropped in, so your processes can look for a file name that matches the constraint, and rename it to not match the constraint.

Still if you can rely on some renaming pattern that will surely not collide or be confused with incoming file names, then it should be possible to rule out race conditions. If two processes both spot the same new file, only one of them can succeed in renaming it (naturally, each process should use its own distinctive pattern for renaming files).

The process that fails when it tries to rename can just go back to looking for another new file.

  • Comment on Re: Synchronizing multiple processes retrieving files out of shared directory

Replies are listed 'Best First'.
Re^2: Synchronizing multiple processes retrieving files out of shared directory
by MitchInOmaha (Initiate) on Mar 20, 2014 at 13:10 UTC
    My original goal was to avoid having filename constraints on the content being dropped off into the shared directory. (Original plan included having multiple customer departments dropping off work, but it's turning out that working entering this directory is all coming from the same location, so we have more control over naming than originally expected.)

    I'm going to start by trying lock files with exclusive locks on them. If a process can't create a new file with an exclusive lock, then they move on to the next file in the directory. If that doesn't work, I'm going to see if I can implement a rename-based process.

    Thanks.

[reply]
      I think if you were to create sibling or sub-directories on the same disk volume as the one where the depository directory exists, then the perl-internal rename function will still work (and still be atomic in the same way). Each process could create its own sub- or sister directory, and the same logic should apply.
[reply]
[d/l]
        if you were to create sibling or sub-directories on the same disk volume as the one where the depository directory exists, then the perl-internal rename function will still work (and still be atomic in the same way).

        Is that true for NFS mounted volumes?

        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

In Section Seekers of Perl Wisdom