Clear questions and runnable code get the best and fastest answer |
|
PerlMonks |
Re: Re: Parallel Processing, Queueing and Schedulingby Anonymous Monk |
on Mar 12, 2003 at 00:53 UTC ( [id://242225]=note: print w/replies, xml ) | Need Help?? |
If the jobs are significant pieces of time, the trick that I have used for this is to store information about what jobs are needed in a database. Then let each machine open up a database connection, open a transaction, figure out which job to do, and then mark it as started. It should issue regular updates if desired. Then when it finishes it marks the job as done. Users have a tool that allows them to add jobs to the database. I didn't develop this into anything complex, but it wasn't hard to get to a usable state. And since coordination is handled in a lightly loaded database, this should scale to a very large number of machines. And it can coordinate processes that need cross-platform resources. Including human intervention! Other solutions worth considering are standard clustering technologies like http://www.mosix.org/, and various solutions that fall under the name grid computing.
In Section
Seekers of Perl Wisdom
|
|