The better answer that I have used in the past for loosely-coordinated multi-processing scripts is to use the filesystem. I will presume that neither the data nor status file reliably arrives first, so we must detect when a pair of files is present, lock the pair, and then proceed with processing.

The simple solution (assuming all workers are running on the same node, as network filesystems can screw this up) is to create a third "flag" file, using sysopen with O_CREAT|O_EXCL from Fcntl, the successful creation of which acts as acquiring a lock for that pair. A process that fails to acquire this lock simply moves on to the next pair, confident that another worker has already claimed that pair.

There is a small potential problem here with stale flag files, but that can be remedied by either shutting the worker horde down and cleaning up any flags left (as I have used in the past when I needed this for a quick-and-dirty multi-process solution) or writing the PID into the flag file to allow a single cleanup process to remove any flags left by workers that are no longer running. When cleanup removes a flag, another worker will eventually find that pair and process it. You will need to manually supervise this kind of operation, because the most likely reason for a worker to fail to complete processing is that that pair exposes a bug in the worker code and will reliably crash the worker process.

In reply to Re: Semaphore puzzle by jcb
in thread Semaphore puzzle by jerryhone

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.