I've done similar batch work -- it's just a tuning issue, and it's going to be different for every application that does this sort of work.

When I've done such stuff in the past, I've left a configuration variable in my code, that tells it how many records to push at a time -- that way, I can easily adjust the number to push single records, or to an insanely high number to push all of the records at once, or a number in the middle to do lots of medium sized blocks.

You can then make an educated guess as to what will be best, and adjust if need be down the road, should you need to tune this aspect of the process.

oh -- and as for the mod_perl comment -- I'd probably avoid it for this application -- if you're only going to be making a few calls against it a month, it's not worth keeping resident in memory except for the times when it's being run. _Maybe_ if you're going to run single records, it'll be worth it, but my gut tells me that the costs will outweigh any benefits for the one load.

(if the time to do the bulk processing is more important than the normal processing done by the machine, then maybe mod_perl, but if it were that important, you'd have done the 'don't use HTTP' path, without question.)

In reply to Re: Submitting large data to CGI by jhourcle
in thread Submitting large data to CGI by TedYoung

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.