What you might be experiencing is an artifact of the data transfer mechanism. When using SELECT on large result sets, the DBI::mysql driver, or rather, the underlying C driver, tends to load in the entire result into RAM. I'm supposing this is a probable cause considering that you are working with log files which can tend to get quite large quickly.

In DBI + 'SELECT *' - Memory Use Galore?, I experienced massive memory utilization even though I thought I was retrieving the result set row by row. MySQL's driver, unless instructed otherwise, will retrieve the entire result set and parcel it out to you row by row from the memory buffer.

kschwab was helpful enough to point out that you are expected to set an option on your statement handle which forces incremental transfer:
$sth->{"mysql_use_result"} = 1;
However, this renders that particular database connection unusuable until you finish with that $sth. If you need to retrieve from one and insert into another, make two DB handles, one for the incremental read, and the other for all the INSERTs.

PostgreSQL likely has a different driver mechanism that does not suffer from this particular artifact.

In reply to Re: memory leaks and mysql by tadman
in thread memory leaks and mysql by ralphie

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.