L~R: I would think the two best forms of queues are databases and queue directories. In your case a directory system such as one that sendmail uses may be very valuable. It will let you lock and manage jobs, you can have different level of queue directories such as:

/var/spool/prog/queue-1
/var/spool/prog/queue-2
/var/spool/prog/queue-bad
/var/spool/prog/queue-lowpriority
[download]

and have the queue runners on each queue manage the flow of jobs between queues. You can bulk up the number of queue runners for a particular queue if the load jumps, and set different priorities for previously failed jobs. Locking can be flock based or child file based (like sendmail -- flock is great for 1 step queue processing, file based is ice if the process has multi steps and needs to be inspected on system failure or reboot before the queue file is running again). You can do the same thing in a database, and it may more sense to -- depending on how the jobs are gathered. Either way, you know what to watch for -- you need tight file locking to avoid races.