Hello fellow Monks,

I have a general implementation/design question that I has been troubling me. We are setting up a web-site that will provide a free resource to researchers in our institution. There are three parts to this:

  1. Front end: CGI, accepts and verifies parameters
  2. Wrapper: call external program to do "data processing"
  3. Results display: CGI to format the data

The part that is troubling me is #2. The data-processing can be pretty computationally intensive. I would prefer to "queue" requests, so that they get processed in serial rather than in parallel. It seems that doing things that way would break the underlying model.

The best option I see is as follows:

  1. Front End
  2. Write parameters to text
  3. Check if results finished
  4. Display results

Once the parameters are written to text, a cron job (okay, this is on WinXP so I guess it would have to be a service?) regularly checks the textfile to see if there are new records to be run. If so, it processes them and dumps the results to a text-file.

I'm not a huge fan of the above strategy because, amongst other things, it makes me go and rewrite the wrapper, which is already written in a standalone, non-CGI version. Are there any other standard solutions to this kind of problem?

In reply to Queuing Input for Serial Processing by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.