As wind says, you want to know where your bottleneck is, and it also depends on what you're doing between parsing and insert.

If you're simply reading records and then inserting, you can simply split the file into chunks and process each in a separate forked process (which, on Unix, is probably faster than perl ithreads) each with it's own DB connection.

If you're doing intensive processing, perhaps you can split that into 2 (or more) phases - then you can implement each phase in a separate Perl program and connect them with pipes, the last phase of which will have the DB connection. This is still only using one DB connection. I'm not sure if you need to use multiple connections for MySQL to use multiple backend processes/threads but you could possibly run this 'pipeline' multiple times also.

Another completely unrelated option you can consider is to do all the processing into a flat file, slurp that over to the db machine and use LOAD DATA LOCAL (or whatever it is - that's from memory) to bulk load the data. That will save you a huge amount of database overhead on that sort of volume of data.

In reply to Re: Threading - getting better use of my MP box by aufflick
in thread Threading - getting better use of my MP box by ethrbunny

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.