LOAD DATA will do mass imports. My issue with it is that (I believe that) single errors will cause an entire import to fail.
Sometimes... If the error is in the data-output being inconsistent then maybe. If the error is because the row violates a pkid then again check out REPLACE / IGNORE clauses. PG does not yet have these options and they save a tremendous amount of time.

Do not ever split on \t for a tab separated file. I've done this many times, that solution is stupid and crufty. Set the sep_char in Text::CSV. All of what your doing now, less the chopping of white space can be done by Text::CSV. It will work faster (if it can make use of Text::CSV_XS) AND it will handle the edge cases which you aren't. There is even DBD::CSV (uses csv_sep_char) which will use Text::CSV_XS on the underside, and you can simple select from one table and insert it into SQL!! Zomfg, split is suicide here.


Evan Carroll
I hack for the ladies.
www.EvanCarroll.com

In reply to Re^2: loading csv files into MySQL databse by EvanCarroll
in thread loading csv files into MySQL databse by zimvoter

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.