Thanks for sharing your success with us!

I have seen similar or even greater performance increases, see: Re^3: Efficient way to handle huge number of records? for a more recent benchmark. In this benchmark, the code started on Friday afternoon and runs until early on Sunday morning. Improved code runs in 3 minutes! And that's just for one 1 million row table! This matters and I am very, very sure about it! I mean couple of minutes versus couple of days!

Current version of this code imports 5,740,776 rows into 8 tables and runs in less than 10 minutes on my slow machine. The production machine will run this much faster. In my application, this is "fast enough".

This is a case where there is a LOT of low hanging fruit and its easy to get it if you know how! 20x performance increase is nothing to be sneezed at!

The next level of performance is how the DB deals with indicies. It is faster to create the indicies after the complete table is built. Getting that next level is harder than the first, most important level.

I congratulate you for your success and also reporting back after some months with the results. This helps others and that is what this forum is about.

In reply to Re^3: Batch Upload/Insert – Row wise with Perl DBI by Marshall
in thread Batch Upload/Insert – Row wise with Perl DBI by shree

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.