Work queue is also what I suggest. But don't use a database as the queue. Use something like redis' FIFO queue. You could get fancy and make a priority queue using sets, but sounds like you want straightforward, and I agree.

The producer process puts work on the atomic queue, worker daemons spin and pop off work to do. Sure you could have the worker daemons fork off children to do the work, but as long as you have the atomic queue then you can just have any number of worker daemons checking for work to to do in a loop - so there is no need to get fancy with the worker processes. Redis (and the Perl client) is not the only way to do this, but it's the one I have the most experience with.

As I stated above, don't use a database to serve the queue. You don't have to use Redis, but do-not use a database (terribly inefficient for this type of middleware).

If you wish for the worker process to communicate back to the work producer, you can use a private Redis channel specified in the chunk of work. However, if you want real messaging you will be best to go with something built for that, like RabbitMQ or something similar but lighter weight.

Work can be shoved into the queue by the producer in JSON or some other easily deserialized format; it can include a "private" redis channel or "mailbox" for the worker thread to send a message to the producer or some other listening. You could actually set up a private mailbox scheme so that the initial contact with work on the queue allows the producer and consumer to have any sort of meaningful conversation you wish.

Also note, the 6.x version of redis supports SSL natively and some level of access controls. I'd use them if going over public internet or crossing any sort of untrusted networks.

In reply to Re^2: Need suggestion on problem to distribute work by perlfan
in thread Need suggestion on problem to distribute work by smarthacker67

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.