Well, I said I would be back if I had any news, so here I am.

I've decided to try and build an UDP communication system between my servers, and so far I haven't had any serious issues with this setup.

I chose this approach considering my specific situation, which might not aply to many others using any sorts of IPCs.

I have no need for persisting the queue items between application restarts, so I avoided the I/O activity caused by a file/database-based system. The servers are always on the same network, so UDP is just fine.

Although I think Corion's advice would probably be the most secure approach, it would also mean a lot of aditional work, such as having satellite servers depending on a single master for distributing the work to be done, or maybe set up another system just to elect a new master when the primary goes offline. Which would obviously set up a whole new level of complexity in the application.

So I just chose what I found out to be the most "cost-effective" solution: an ad-hoc, masterless system, where each servers asks (single tiny UDP datagram) its peers if they are working on the item which the server is about to enqueue. From the several runs that I've done, after many tweaks, I've seen duplicated processings at the average rate of 1 for each several thousands.

Since the main problem with duplicated processing, in my case, is the waste of CPU cycles for the servers doing twice the same job (no data loss, corruption or anything like that), it's perfectly fine for me.

In reply to Re^2: Sync item list between perl scripts, across servers by reinaldo.gomes
in thread Sync item list between perl scripts, across servers by reinaldo.gomes

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.