Aside from the other suggestions, might I recommend a serializable workflow object? You would basically draw a work flowchart with a beginning and an end. Each stage is a different "state" in the workflow with a proper commit/rollback procedure. I think Pixie might be great for something like this. Your program could load all workflow objects and either pass each object to a different thread/POE process or just load them sequentially if the load was light enough.

For the "post to form" job, the workflow object would attemp to post the data and, if failing, move itself to the bad queue. For that, you might want a single workflow collection object. You only need (hopefully) one instance and use the flyweight pattern to identify the workflow objects in various "in", "work", "out", and "bad" queues. (I'm just guessing about the flyweight pattern because I think it might make it easier if you're running multiple threads or have workflow objects on another marchine).

Conceptually I think it's a pretty clean model and hopefully would accurately reflect your business needs. It also makes locking simple if you have a single collection object assigning tasks to the workflow objects (theoretically).

Cheers,
Ovid

New address of my CGI Course.

In reply to Re: Managing a web form submission work queue by Ovid
in thread Managing a web form submission work queue by Limbic~Region

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.