You mention that portability is a large concern, so you want to avoid db-specific bulk loaders. Writing a script that would create the various DB specific loader files wouldn't be that difficult and would actually probably be a whole lot more beneficial. I find many uses for the bulk loading abilities of Oracle (and have also found that few people really know the true power and help they can bring to the table). And the performance gains you get with these utilities are HUGE. We are not talking just 10-20%, but in many cases, hundreds to thousands of times faster than normal inserts.

Also, if you are going to stay with the standard insertion method:

Threads are mentioned, but if there are any constraints set up on the table then you'll lose any performance increase on the inserts as the db has to deal with checking constraints on two sets of insertions, not a single stream of them.

500 rows between commits may be something that you can tweak (either larger or smaller) to see additional performance gains based on the performance and memory of your current DB.

In reply to Re: Implementing a buffered read-and-insert algorithm by RiotTown
in thread Implementing a buffered read-and-insert algorithm by radiantmatrix

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.