Re^3: Musings about a database ecology

but it needs to be a system that can guarantee atomic operations, and survive a restart

I would want such a system, but talexb hasn't indicated if either of those are needed. Plus, you don't need atomicity if you only have one consumer. If the daemon is single-threaded, atomicity is undesired overhead.

A database of some kind, even if it's just Cache, is definitely needed at some layer. I would definitely avoid the relational overhead if it's just a queue of discrete atoms.

But, I think that saying "It needs a database" and "It must use a database as the primary queue implementation" are two separate statements. Flushing to a database is definitely important, to make sure that restarts are survivable¹. However, I would use a Perl AoH as the queue.

Of course, survivable is a spectrum. I would initially propose flushing to the datastore every minute or so. So, you would have up to 60 seconds of data that could be lost. Depending on the system, this may be ok or it may not be ok. (In some systems, it might even be desirable, depending on your other components.)

Being right, does not endow the right to be rude; politeness costs nothing.
Being unknowing, is not the same as being stupid.
Expressing a contrary opinion, whether to the individual or the group, is more often a sign of deeper thought than of cantankerous belligerence.
Do not mistake your goals as the only goals; your opinion as the only opinion; your confidence as correctness. Saying you know better is not the same as explaining you know better.

Comment on Re^3: Musings about a database ecology

Replies are listed 'Best First'.
Re^4: Musings about a database ecology by perrin (Chancellor) on Dec 20, 2004 at 20:47 UTC
For me, the bottom line is that storing a queue in a networked database works and scales well, and requires very little development. I wouldn't look at harder, more customized solutions unless I was doing something that really fit poorly into the existing networked databases.	[reply]