L~R: I would think the two best forms of queues are databases and queue directories. In your case a directory system such as one that sendmail uses may be very valuable. It will let you lock and manage jobs, you can have different level of queue directories such as:

/var/spool/prog/queue-1
/var/spool/prog/queue-2
/var/spool/prog/queue-bad
/var/spool/prog/queue-lowpriority

[download]
and have the queue runners on each queue manage the flow of jobs between queues. You can bulk up the number of queue runners for a particular queue if the load jumps, and set different priorities for previously failed jobs. Locking can be flock based or child file based (like sendmail -- flock is great for 1 step queue processing, file based is ice if the process has multi steps and needs to be inspected on system failure or reboot before the queue file is running again). You can do the same thing in a database, and it may more sense to -- depending on how the jobs are gathered. Either way, you know what to watch for -- you need tight file locking to avoid races.


-Waswas

In reply to Re: Managing a web form submission work queue by waswas-fng
in thread Managing a web form submission work queue by Limbic~Region

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.