I went through the same learning curve recently.

Lesson learned the hard way: Do not do mlutiple inserts unless you have all day to wait...

Use "LOAD DATA INFILE ". This is an sql utility specifically for importing csv data into tables. It's very very fast. Example? My linewise-insert programming that strangely ressembled yours was working at ~300 file lines per hour with a complex line-analysis intermediate step on a ~1.5m line file (650Mb). Once I had completed the intermediate line-wise analysis with another program so the data was pre-formatted ready for importation, it took about a half hour for LOAD DATA INFILE to import it all.

In your case this is going to be childishly simple, since it appears you have only one table to insert to: 

        $sql = 'LOAD DATA INFILE $filename.csv INTO TABLE T_$;';

[download]
There are options available for field terminators and line terminators. Check out

http://dev.mysql.com/doc/mysql/en/LOAD_DATA.html

Forget that fear of gravity, Get a little savagery in your life.

In reply to Re: Implementing a buffered read-and-insert algorithm by punch_card_don
in thread Implementing a buffered read-and-insert algorithm by radiantmatrix

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.