The problem is less the complexity, it's more the application that at least I lack. I have some small toy code that essentially does the following:

1. Connect to another machine (via ssh, rsh or telnet)
2. Spawn perl -x there
3. Pipe some small code there that does MIME-ish decoding of stuff read from STDIN and outputs MIME-ish delimited Data::Dumper elements to STDOUT

With that code (about 50 lines or so), I get a remote Perl interpreter to which I can easily pass code to execute. The main uglyness is the passing of parameters back, as you most likely want to produce output (array elements) as soon as they become available and not when the remote function has returned completely.

The second ugly problem is of course the collecting and coordination of multiple such processes, and for that, I would either resort to a database for serialization (thus ditching my low-infrastructure approach) or have to resort to POE for any but the simplest tasks, neither of which is really nice.

I originally developed that code to implement a perlish rsync variant, so I only need some more or less synchronous execution, but the packaging of input and output parameters and an elegant method of actually writing the remote code is what paused the development. Using B::Deparse lets me transfer the trivial code over to the remote machine, so Perl catches syntax errors on the local side already, but what remains and stops me is some way of transferring data more efficiently via pack/unpack instead of Data::Dumper.

Update: Added last paragraph depicting the sad state of this idea

Personally, rather than a central application parcelling data up, dispatching it to one of a list of known workers, then polling the workers for their results before retrieving them, and dispatching the next batch, I'd turn things around. Use the web server model--and a websever too.

Workers hit one page to get data, and another page to return results. The central controller can be any webserver you like from Apache2 to HTTP::Daemon.

The workers could use LWP to fetch the work_to_do. The page they fetch would contain the data to process and even an embedded code section to use to process it, plus a form on which to return the results. That easily allows for multiple uses and clean segregation of the returned data as the code comes from the page called and the processed data is dispatched back to the address referenced by the form.

Testing can be done by accessing the server using a standard web browser and pasting "processed data". If the application needs to deal with large volumes of return data, use POST instead of GET.

The security of the application can use the standard webserver authentication mechanisms. Assuming that the server is properly controlled and vetted, I'm assuming a LAN/intranet environment, then the code segments should be as secure as the server is. You can even use https: to provide encryption if required.

IBM DeveloperWorks has a large, multiplatform section on grid computing. You may be interested in the tutorial "Build a grid with Perl". Registration is required, but it's free.

-- Rocco Caputo - http://poe.perl.org/

One piece of advice, slapping security on it as an afterthought looks like a bad idea to me. I'd solve the security implications first, and intergrate them with the solution.

• Take a large job and divide it into N small jobs
• A central (or distributed) system for keeping track of job status. This could be database tables and/or a central process.
• Queues for submitting jobs to nodes and taking appropriate action when they are completed. If results are stored centrally in a database, this might mean only monitoring node status and tracking errors as well as completion of jobs.

A commercial product called LSF (www.platform.com) does much of this and works well with cluster applications. With it, one may create queues for specific systems or sets of systems, submit jobs, and monitor them.

For grid versus cluster applications, all this gets more fuzzy. Are the grid systems shared? How reliable are they? How much redundancy is needed?

Like I said, these are some thoughts and questions. Hope it's useful. :)