Personally, I would only keep one boss thread that feeds the workers. As you seem to have the filesystem shared between the machines, you can keep your queue either as a file or have the second machine connect via TCP to the boss thread to read available files from it.

Having two boss threads scan the same directory is a recipe for disaster, or at least for lots of interesting failure scenarios.

My favourite approach is to organize the queues as directories and have the workers move the files between the directories:

    /work/incoming
    /work/processing
    /work/done

[download]

This will be problematic if your (network) filesystem does not support atomic rename. I think that NFS does not support atomic rename.

If your filesystem supports atomic append, you can simply have each worker append a line with its machine name, pid and the filename to one common file and then reread the file to see if another thread snatched the file before it. This would mean that the file grows, but you can either move that file away or just truncate it from time to time. Truncating would mean that files get processed twice, which might not be acceptable.

In reply to Re: Sync item list between perl scripts, across servers by Corion
in thread Sync item list between perl scripts, across servers by reinaldo.gomes

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.