I'm going to talking at YAPC in a couple of weeks about techniques for website scalability. One of the things I'll be discussing is the use of job queuing to handle large numbers of requests for a limited resource. My main example will be what Ticketmaster does on their site. However, the Ticketmaster code for this is tightly coupled to their backend systems and is thus not open source. I want to give people some open source examples to look at, so I'm wondering what others have been using.

Ideally I am looking for something with these characteristics:

• works across a cluster of machines
• relatively easy to administer
• provides control over how many jobs are running at once
• allows asynchronous calls, i.e. a client sends in a request and then can check on the status and eventually the results later on

Here are the ones that I've turned up. Any suggestions of better ones, or feedback on the effectiveness of these tools would be useful.

Spread::Queue

This is the one that currently seems best to me. It uses the Spread toolkit to handle the communications between clients, a queue manager process, and worker processes.

Parallel::PVM

PVM seems like a kind of old-school system built for running scientific applications on clusters. It's looking a little long in the tooth, but I have not tried it yet.

Parallel::MPI and Parallel::MPI::Simple

MPI is basically a newer replacement for PVM, but still more oriented towards distributed processing for science apps.

IBM's grid stuff

I need to look into this one more. It seems to use some more modern protocols (web services) but other than that it's unclear what advantage this might have over PVM and MPI.

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.