If database insert speed is the bottleneck, and you have clean data, one of the best ways to gain speed is to drop indexes and constraints on the table before you insert, then re-create them afterwards.

This presumes, of course, that you have permission to execute DDL in the database, that you know the table schema, that you won't hose up your data by not having constraints, and so on.

Also, some databases, like Sybase and SQL Server, have specialized bulk-insert interfaces that will allow you to load data irrespective of constraints. But that may not work for you, since you seem to indicate you're going against multiple platforms at different times.

Do not rebuke them with harsh words ... but rather lead them gently - with URLs - so that they may learn wisdom.

In reply to Re: Implementing a buffered read-and-insert algorithm by VSarkiss
in thread Implementing a buffered read-and-insert algorithm by radiantmatrix

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.