If the jobs are significant pieces of time, the trick that I have used for this is to store information about what jobs are needed in a database. Then let each machine open up a database connection, open a transaction, figure out which job to do, and then mark it as started. It should issue regular updates if desired. Then when it finishes it marks the job as done.

Users have a tool that allows them to add jobs to the database.

I didn't develop this into anything complex, but it wasn't hard to get to a usable state. And since coordination is handled in a lightly loaded database, this should scale to a very large number of machines. And it can coordinate processes that need cross-platform resources. Including human intervention!

Other solutions worth considering are standard clustering technologies like http://www.mosix.org/, and various solutions that fall under the name grid computing.

In reply to Re: Re: Parallel Processing, Queueing and Scheduling by Anonymous Monk
in thread Parallel Processing, Queueing and Scheduling by submersible_toaster

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.