Re: what do you use for job queuing?

I've used a sendmail / IMAP setup for this before. It's an excellent way of providing a persistent distributed routable queuing architecture, with a well known and documented API for handling complex messaging. A lot of people give you quizzical looks when you explain to them how to set this sort of thing up, but then again, it's how "job queuing" is done if the endpoints are people, so why wouldn't it scale splendidly to programs? Assign different processes different email addresses, write a procmail file writer to help you do load balancing if need be accross a cluster Built right in you get point to point and message list distribution, even request receipts and complex attachment hierarchies if you so chose :) It's my favorite solution to this particular problem.

Comment on Re: what do you use for job queuing?

Replies are listed 'Best First'.
Re^2: what do you use for job queuing? by perrin (Chancellor) on Jun 04, 2004 at 17:58 UTC
I agree that e-mail offers some attractive features in terms of queueing messages and handling temporary delivery failures robustly, and I have heard people talk about doing it this way before. I would say it mostly just solves the message queue issue though. It might have issues with frequent polling, just like a database-backed queue would, although I suppose an e-mail server is built to do that well. It would also be difficult to get the current status of a job, but maybe "received" is good enough for most cases. I think you would end up needing to build some extra infrastructure to allow a client to check for the response to a specific job, or is there a simple way to do that with IMAP queries and subject lines or something? Also, how would you handle distributing messages across a set of worker processes? Would you use e-mail for that too, sending messages on from a dispatcher to specific workers with unique e-mail addresses?	[reply]

Replies are listed 'Best First'.

Re^2: what do you use for job queuing?
by perrin (Chancellor) on Jun 04, 2004 at 17:58 UTC

It might have issues with frequent polling, just like a database-backed queue would, although I suppose an e-mail server is built to do that well. It would also be difficult to get the current status of a job, but maybe "received" is good enough for most cases.

I think you would end up needing to build some extra infrastructure to allow a client to check for the response to a specific job, or is there a simple way to do that with IMAP queries and subject lines or something? Also, how would you handle distributing messages across a set of worker processes? Would you use e-mail for that too, sending messages on from a dispatcher to specific workers with unique e-mail addresses?

[reply]