in reply to Queuing system for running a perl program

On a more general level, this sort of thing is called a batch-job monitoring system, or maybe a workload scheduler, and there is already a lot of “prior art” out there on the Internet, ready for the taking.   However, this is one of the most-reinvented wheels, because most of the time people just homebrew something up ... without considering, for example, the issues related to enabling a cluster of computers to reliably process the work given that none of them might be totally reliable.

The usual “homebrew” beginning starts with an SQL table (in a database system that supports transactions), and a cron job – or simply worker daemons who sleep() – who read that table as a queue of work-to-do.   You homebrew a user-interface screen that queries the table to tell users when the work is done.   (They hit Refresh periodically.)   The worker daemons retrieve work from the table, using transactions to avoid conflict with one another, and they post completion status when they are done.   The number of daemons dictates the number of units of work that can proceed in parallel.   The web-page or what-have-you is only a job-entry and job-observation system, as well as perhaps the way that completed output is retrieved for use.

It’s a bit strange to me that, whereas in the earlier mainframe days all work was done with batch jobs, when Un*x came along with its champion of “interactivity,” it never acquired a “standard” background-processing tool other than cron.   Which is woefully insufficient.   If, for example, that “extra standalone software” which is “time-consuming” could also have associated resource-conflicts, a simplistic solution will either constantly-overcommit because it does not know about these limits (and thus, run slower than it could, or not at all); or, it will fail to initiate work as fast as it could because it does not know when more work could be started (somewhere) and it is not aware of multiple machines..   If there is a possibility that a unit-of-work might “abnormally end” (ABEND, as we used to say), a simplistic solution might not be able to recover at all.   When you start adding these to your homebrew, the re-invented wheel just gets bigger and bigger.   Therefore, I would canvass the Internet to look at available pre-existing alternatives, including those that are commercial products.   None of these will care if the workload is or isn’t Perl.