Re^2: Reliable Work Queue Manager

I tend to agree that a database is the way to approach this, but there are two ways I can think of that avoid installing a 3rd-party DB for what might be a very simple queue.

The first suggestion would be to implement said queue using DBD::SQLite2 via DBI. This path would allow an easy upgrade to a full-scale RDBMS in the future.

Then there's DBM::Any. DBM files could work well for this if you plan carefully, and thanks to DBD::DBM, one could ensure a reasonable path to a full RDBMS in the future.

<-radiant.matrix->
A collection of thoughts and links from the minds of geeks
The Code that can be seen is not the true Code
"In any sufficiently large group of people, most are idiots" - Kaa's Law

Comment on Re^2: Reliable Work Queue Manager

Replies are listed 'Best First'.
Re^3: Reliable Work Queue Manager by ph713 (Pilgrim) on Oct 27, 2005 at 18:20 UTC
Having a truly transactionally coherent ACID relational database at your disposal of course makes things like persistent jobs queues a breeze. Then you just need a "jobs" table with an autoincrementing serial number for a key and you're pretty much done. But I'd be careful of thinking that using DBM-based solutions buys you the same gaurantees as a real database, I would imagine there are a lot of corner cases for recovery that don't work out so well in certain failure scenarios. But, IMHO, relying on having a full-blown RDBMS installed anwhere you use your queueing library isn't very good planning either. A lot of good n-tier application architectures can be built assuming you have a disk-persistent network-transparent queueing system in place that you can run on arbitrary machines without the support of an RDBMS or specific dbm implementation, along the lines of IBM MQseries in terms of functionality. You don't want to have to put local RDBMS's on every node involved in the architecture in order for it to work. My opinion is that for a general-purpose queueing module one should build a solution based on queue directory management using whatever atomic and/or locking resources you have available. POSIX specifies atomic rename() and good fcntl() locks. Then you can deploy the module in any reasonable environment which implements the POSIX primitives you depend on (excuse my *nix centricity, I don't take Windows seriously at all, and I'm not really sure whether it supports POSIX in that regard). Come to think of it, sendmail/postfix + (E)SMTP amounts to a disk-persistent network-transparent job queueing system using POSIX primitives and a queue directory on the local machines. It makes a fine example, spam problems notwithstanding :)	[reply]

Replies are listed 'Best First'.

Re^3: Reliable Work Queue Manager
by ph713 (Pilgrim) on Oct 27, 2005 at 18:20 UTC

Having a truly transactionally coherent ACID relational database at your disposal of course makes things like persistent jobs queues a breeze. Then you just need a "jobs" table with an autoincrementing serial number for a key and you're pretty much done.

But I'd be careful of thinking that using DBM-based solutions buys you the same gaurantees as a real database, I would imagine there are a lot of corner cases for recovery that don't work out so well in certain failure scenarios.

But, IMHO, relying on having a full-blown RDBMS installed anwhere you use your queueing library isn't very good planning either. A lot of good n-tier application architectures can be built assuming you have a disk-persistent network-transparent queueing system in place that you can run on arbitrary machines without the support of an RDBMS or specific dbm implementation, along the lines of IBM MQseries in terms of functionality. You don't want to have to put local RDBMS's on every node involved in the architecture in order for it to work.

My opinion is that for a general-purpose queueing module one should build a solution based on queue directory management using whatever atomic and/or locking resources you have available. POSIX specifies atomic rename() and good fcntl() locks. Then you can deploy the module in any reasonable environment which implements the POSIX primitives you depend on (excuse my *nix centricity, I don't take Windows seriously at all, and I'm not really sure whether it supports POSIX in that regard).

Come to think of it, sendmail/postfix + (E)SMTP amounts to a disk-persistent network-transparent job queueing system using POSIX primitives and a queue directory on the local machines. It makes a fine example, spam problems notwithstanding :)

[reply]