What you might be experiencing is an artifact of the data
transfer mechanism. When using SELECT on large result sets,
the DBI::mysql driver, or rather, the underlying C driver,
tends to load in the entire result into RAM. I'm supposing
this is a probable cause considering that you are working
with log files which can tend to get quite large quickly.
In
DBI + 'SELECT *' - Memory Use Galore?, I experienced massive memory utilization
even though I thought I was retrieving the result set row
by row. MySQL's driver, unless instructed otherwise, will
retrieve the entire result set and parcel it out to you
row by row from the memory buffer.
kschwab was helpful enough to
point out that you are
expected to set an option on your statement handle which
forces incremental transfer:
$sth->{"mysql_use_result"} = 1;
However, this renders that particular database connection
unusuable until you finish with that $sth. If you need to
retrieve from one and insert into another, make two DB handles,
one for the incremental read, and the other for all the
INSERTs.
PostgreSQL likely has a different driver mechanism
that does not suffer from this particular artifact.