in reply to memory leaks and mysql

What you might be experiencing is an artifact of the data transfer mechanism. When using SELECT on large result sets, the DBI::mysql driver, or rather, the underlying C driver, tends to load in the entire result into RAM. I'm supposing this is a probable cause considering that you are working with log files which can tend to get quite large quickly.

In DBI + 'SELECT *' - Memory Use Galore?, I experienced massive memory utilization even though I thought I was retrieving the result set row by row. MySQL's driver, unless instructed otherwise, will retrieve the entire result set and parcel it out to you row by row from the memory buffer.

kschwab was helpful enough to point out that you are expected to set an option on your statement handle which forces incremental transfer:
$sth->{"mysql_use_result"} = 1;
However, this renders that particular database connection unusuable until you finish with that $sth. If you need to retrieve from one and insert into another, make two DB handles, one for the incremental read, and the other for all the INSERTs.

PostgreSQL likely has a different driver mechanism that does not suffer from this particular artifact.

Replies are listed 'Best First'.
Re: Re: memory leaks and mysql
by ralphie (Friar) on Jul 02, 2001 at 16:33 UTC
    thanks - that's very helpful.