-- Randal L. Schwartz, Perl hacker
Be sure to read my standard disclaimer if this is a reply.

• open source
• provides a tcl ( I know, I know ) interface where you can even write your own scheduler
• stdout/stderr watching, as you mentioned
• an open api for reporting status back
• dependency chain ( only run job b if job a succeeds, etc)

Here's the blurb from their website:

The Portable Batch System (PBS) is a flexible batch queueing and workload management system originally developed for NASA. It operates on networked, multi-platform UNIX environments, including heterogeneous clusters of workstations, supercomputers, and massively parallel systems. Development of PBS is provided by Altair Grid Technologies.

UPDATE: ahhh, after seeing merlyn's recommendation of POE i changed my mind. Definitely give POE a try for this problem.

jeffa

L-LL-L--L-LL-L--L-LL-L--
-R--R-RR-R--R-RR-R--R-RR
B--B--B--B--B--B--B--B--
H---H---H---H---H---H---
(the triplet paradiddle with high-hat)

maybe Overkill, and no MS-Win support (but it might work with cygwin type environment) is PBS. it's widely used in I2/Globus/Grid projects. you would still need to do your template part, but PBS can schedule/manage the jobs across available nodes. i don't use it myself, but we have a (ew, 600ish nodes, 2-4 CPUs/node) cluster that's booked for the next year or so managed by the PBS stuff.

IRIX and Linux are on the supported list (with other *NIX flavors as well).

Update:Dammit , I have think I've screwed up my terminology again and confused the issue.
I will try to describe it better.
A number of machines exist, running a client that offers that machine's CPU as available to use.
On seeing an available client, the server determines which task in the queue has first crack at the CPU resource, and sends the client a segment of that task.
The server is listening constantly to other clients, regarding their progress.
Users have a seperate client tool to drop tasks on the queue, and monitor their progress

More to the point , tasks are being processed by plain old executables that know nothing nor need to know - ie host A does not care that while it processes parts 1-10, Host B is processing parts 11-20 of the same task.

Update:Firstly I would just like to say thankyou to everyone for your ideas and I will of course be following up and researching many of these suggestions. If you have not been ++'d by me in this thread yet, when the vote-fairy returns it will be so.

Secondly, I will be taking a clue-by-four to my silly self, as my requirements are going to have to change. Supporting all those platforms is going to create more work to achieve than it will leverage in processing grunt. Powers that be are already making linux noises RE 3d Animation applications, so supporting win32 AND *n(u|i)x flavors is biting off more than I can chew. Did I point out that PHB are no prepared to buy any more hardware yet to aid processing, and are less inclined to spend money on software to manage the queue - particulary because there isn't much hardware for it to manage (it gets even more circular after that so I'll stop now).

Hi,

I used to work for a fiber optics lab at a University. We had many users vying for control of the computational cluster, and needed a way to divide time among them based on time, project priority, etc, etc.

I ended up going with Condor. Condor will support most of what you're asking for. It will also run MPI and PVM jobs, so you can integrate parallel-processing jobs into the system. It's not difficult to setup (does require a decent sysadmin), runs on Unix and Windows, and seems to have a decent userbase. I was able to get answers from some of the developers when I emailed.

Good luck,
ibanix

$ echo '$0 & $0 &' > foo; chmod a+x foo; foo;

If the jobs are significant pieces of time, the trick that I have used for this is to store information about what jobs are needed in a database. Then let each machine open up a database connection, open a transaction, figure out which job to do, and then mark it as started. It should issue regular updates if desired. Then when it finishes it marks the job as done.

Users have a tool that allows them to add jobs to the database.

I didn't develop this into anything complex, but it wasn't hard to get to a usable state. And since coordination is handled in a lightly loaded database, this should scale to a very large number of machines. And it can coordinate processes that need cross-platform resources. Including human intervention!

Other solutions worth considering are standard clustering technologies like http://www.mosix.org/, and various solutions that fall under the name grid computing.

Openmosix contains several improvements by Moshe Bar on the original work of Prof. Barak (it starteted as fork to continue this open project as gpl) try it if you want to use a mosix type linux kernelpatch for clustering :)

I don't know if this helps as an idea, but on a recent work-related project, I wrote a wrapper that checked a database queue for waiting tasks (in my case, account provisioning items), changed the status to pending, processed the item (either itself or through the use of a helper application), then changed it again to a completed or failed status. I had the luxury of having each wrapper only look for one type of item, though, where you would have to do a query probably based on some priority rating or something.

Lots of good suggestions already, but good luck, and hope the idea helps.

We use Sun's Grid Engine on Red Hat 7.2 and 8. It works very well.

I'll second this one. Like most offerings from Sun, It's amazing general, and the docs skimp on a few things, but it works quite well. It also has clients for several different OSes, and source code is available here at http://gridengine.sunsource.net/ --Hawson