Re: Parallel Processing, Queueing and Scheduling

Update:Dammit , I have think I've screwed up my terminology again and confused the issue.
I will try to describe it better.
A number of machines exist, running a client that offers that machine's CPU as available to use.
On seeing an available client, the server determines which task in the queue has first crack at the CPU resource, and sends the client a segment of that task.
The server is listening constantly to other clients, regarding their progress.
Users have a seperate client tool to drop tasks on the queue, and monitor their progress

More to the point , tasks are being processed by plain old executables that know nothing nor need to know - ie host A does not care that while it processes parts 1-10, Host B is processing parts 11-20 of the same task.

I apologise again for the confusion, the -- storm has already begun :(

Update:Firstly I would just like to say thankyou to everyone for your ideas and I will of course be following up and researching many of these suggestions. If you have not been ++'d by me in this thread yet, when the vote-fairy returns it will be so.

Secondly, I will be taking a clue-by-four to my silly self, as my requirements are going to have to change. Supporting all those platforms is going to create more work to achieve than it will leverage in processing grunt. Powers that be are already making linux noises RE 3d Animation applications, so supporting win32 AND *n(u|i)x flavors is biting off more than I can chew. Did I point out that PHB are no prepared to buy any more hardware yet to aid processing, and are less inclined to spend money on software to manage the queue - particulary because there isn't much hardware for it to manage (it gets even more circular after that so I'll stop now).

I am once again reminded why Perlmonks is the first page I open when I arrive at work. Thanks again.
-toaster.

I can't believe it's not psellchecked

Comment on Re: Parallel Processing, Queueing and Scheduling

Replies are listed 'Best First'.
Use Condor by ibanix (Hermit) on Mar 12, 2003 at 01:19 UTC
Hi, I used to work for a fiber optics lab at a University. We had many users vying for control of the computational cluster, and needed a way to divide time among them based on time, project priority, etc, etc. I ended up going with Condor. Condor will support most of what you're asking for. It will also run MPI and PVM jobs, so you can integrate parallel-processing jobs into the system. It's not difficult to setup (does require a decent sysadmin), runs on Unix and Windows, and seems to have a decent userbase. I was able to get answers from some of the developers when I emailed. Good luck, ibanix `$ echo '$0 & $0 &' > foo; chmod a+x foo; foo;`	[reply] [d/l]
Re: Re: Parallel Processing, Queueing and Scheduling by Anonymous Monk on Mar 12, 2003 at 00:53 UTC
If the jobs are significant pieces of time, the trick that I have used for this is to store information about what jobs are needed in a database. Then let each machine open up a database connection, open a transaction, figure out which job to do, and then mark it as started. It should issue regular updates if desired. Then when it finishes it marks the job as done. Users have a tool that allows them to add jobs to the database. I didn't develop this into anything complex, but it wasn't hard to get to a usable state. And since coordination is handled in a lightly loaded database, this should scale to a very large number of machines. And it can coordinate processes that need cross-platform resources. Including human intervention! Other solutions worth considering are standard clustering technologies like http://www.mosix.org/, and various solutions that fall under the name grid computing.	[reply]
Re: Re: Re: Parallel Processing, Queueing and Scheduling by maksl (Pilgrim) on Mar 12, 2003 at 22:02 UTC
Openmosix contains several improvements by Moshe Bar on the original work of Prof. Barak (it starteted as fork to continue this open project as gpl) try it if you want to use a mosix type linux kernelpatch for clustering :)	[reply]
Re^2: Parallel Processing, Queueing and Scheduling by atcroft (Abbot) on Mar 12, 2003 at 05:18 UTC
I don't know if this helps as an idea, but on a recent work-related project, I wrote a wrapper that checked a database queue for waiting tasks (in my case, account provisioning items), changed the status to pending, processed the item (either itself or through the use of a helper application), then changed it again to a completed or failed status. I had the luxury of having each wrapper only look for one type of item, though, where you would have to do a query probably based on some priority rating or something. Lots of good suggestions already, but good luck, and hope the idea helps.	[reply]


laziness, impatience, and hubris
	PerlMonks