Um, slightly off-topic here, but when you said:
... this usually didn't matter because MySQL was able to handle quite a lot of inserts in a single transaction. That is, until it couldn't, and blew up with a buffer size error in the middle of an 8-hour job.

I had to wonder: are you sure you want to be doing that many inserts via DBI? If, instead, you could save the insertion data to a file, and when the file is complete and closed, you execute a single "LOAD DATA INFILE" statement, not only would you vastly reduce the unpleasant likelihood of bad things happening in the middle of database modifications, but also a DBI process that currently takes 8 hours might end up taking noticeably less time.

(Of course, I could envision situations where other database things need to be done that depend on, and must be interleaved with, a sequence of inserts. But in a case like that, I'd still hope to find a way to refactor the task so that inserts can be done in bulk with the db-server's native data-import tools. And then there might be the problem of permissions, if the server is running remotely and being managed by others, in which case some diplomacy might be worth trying...)

In reply to Re: The trap of reference numification by graff
in thread The trap of reference numification by samtregar

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.