comment on

I extended an existing system based on XML RPC to do something similar (see the Frontier::Daemon and Frontier::Client modules for more info on setting up an XML RPC system). Using that approach, you could do something like this:

Create a centralized queue server that accepts and queues jobs to be run. A system to track the completion of jobs could be built into this server. In addition, you'll need a system to track which machines are available and which are busy (the max number of running jobs allowed, as defined by you).
When a job is completed, an XML RPC message can be sent back to the job tracking server.

There are probably much more efficient ways to approach this, but the XML RPC system worked well in our case because it fit into our current system (we already had one set up) and it was extremely easy to implement during refactoring. Specifically, data structures can be passed as parameters without having to process and pass them on the command line or writing them to a data file. Those parameters could include data, input/output filenames, the number of machines to run on, the priority of the job, etc.

If you need more complex communication (IPC, for example), XML RPC is probably not the best choice.

The gory details:

We set up our system this way:

We wrote a class to track machine availability.
We wrote a class to represent instances of jobs and maintain a job queue. The objects contain all data needed to run the job - program name, parameters, etc.
The centralized XML RPC server accepts job requests from external processes, creates job objects out of them, and looks for available machines. If a machine is available the job is run, otherwise the job is added to the queue.
When a job completes, it sends a message back to the centralized server that contains exit information. The machine availability is adjusted accordingly.

This description is actually a simplification of our system - in reality we have two other RPC servers listening for requests from the jobs that are being executed.

Keep in mind that when a message is sent to an XML RPC server, the server waits until that message (or more accurately, the sub called by it) is finished processing before it accepts another message. In other words, the server queues up RPC requests while it processes. If you want a nonblocking server (as we did), you need to set up your own queue system so the server can send a return message to the caller immediately. This allows some jobs to be queued and executed sequentially, while at the same time allowing other jobs to be executed immediately.

I'm sure more experienced and savvy monks will know of more efficient ways to approach this problem. I look forward to reading their responses.

In reply to Re: Distributed Computing Anyone? by bobf
in thread Distributed Computing Anyone? by mykbaker

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		`&`
	<		`<`
	>		`>`
	[		`[`
	]		`]`

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.