This is driving me insane...

I have a perl script that digests some info and loads it into mysql and it was running happily, albiet a bit slowly, inserting line by line using DBI.

While researching mysql optimazation I found out that I should be writing the data to a file and bulk loading it with "load data infile...".

When I got that running, I found that I getting 3557 records as opposed to the 3599 I was expecting and the last record was monged (all defaults or nulls). I check the tmp file and it had 3599 lines, I printed out the command from the perl script and appened it to mysql -e and got 3599 records.

I thought, maybe, it was the pipe not clearing before the script moved on, so I tried to find out how flush things, but couldn't find anything. I even tried running a commit...

I slept on it and this morning decided that since it works from the command line, I'd just rip out the DBI stuff and run the command line version with back ticks... 3557 records!

ARGHHHHHHHHHHHHHHHHH!

I saw something simular in "Unable to load entire CSV file into DB", but didn't see a resolution or any ideas I though applied to me...

Can anyone help me, before I go insane?

Thanks,

Jimbus

Never moon a werewolf!

In reply to DBI/load data infile issue by jimbus

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.