in reply to Out of Memory selecting from MySQL

Somni:
Actually, I'm cleaning up the data before adding it to the final array.
Thx anyway

BrowserUk
I thought it might be overhead, but I didn't realise there was that much!
I've gone with batching 100k rows at a time
Goes fast enough (for now), but I'll keep the pre-size thing in mind for future ref.

This is on a prod box with 8GB RAM...

Cheers
Chris

Replies are listed 'Best First'.
Re^2: Out of Memory selecting from MySQL
by BrowserUk (Patriarch) on Oct 12, 2007 at 06:56 UTC
    I've gone with batching 100k rows at a time Goes fast enough (for now), but I'll keep the pre-size thing in mind for future ref.

    Even with 100k rows at a time, pre-sizing the array would probably show a measurable performance improvement due to the removal of the need for doubling and copying. And it won't cost you anything.

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

In Section Seekers of Perl Wisdom