Re^4: Reducing Memory Usage

Replies are listed 'Best First'.
Re^5: Reducing Memory Usage by BrowserUk (Patriarch) on Jul 16, 2004 at 08:59 UTC
Ouch! It's not your data, but your objects that are killing you. I just loaded 2_000_000 objects ( hash-based, each containing 1 key - scalar 8 bytes. 1 value - string 80 bytes). I put the 2_000_000 object handles into a hash keyed by the id. Total data 2M * 80 = 152 MB. Total memory consumed 700MB. Add a few nested objects in each of those objects and you can see where your memory is going to. Imagine if perl stored unicode as objects, 1 character per, and strings were arrays of theose objects. A convenient OO-schema, but with today's hardware and today's memory limits, totally impractical. There has to be a better way. Examine what is said, not who speaks. "Efficiency is intelligent laziness." -David Dunham "Think for yourself!" - Abigail "Memory, processor, disk in that order on the hardware side. Algorithm, algoritm, algorithm on the code side." - tachyon	[reply]

Replies are listed 'Best First'.

Re^5: Reducing Memory Usage
by BrowserUk (Patriarch) on Jul 16, 2004 at 08:59 UTC

Ouch! It's not your data, but your objects that are killing you.

I just loaded 2_000_000 objects ( hash-based, each containing 1 key - scalar 8 bytes. 1 value - string 80 bytes). I put the 2_000_000 object handles into a hash keyed by the id.

Total data 2M * 80 = 152 MB.

Total memory consumed 700MB.

Add a few nested objects in each of those objects and you can see where your memory is going to.

Imagine if perl stored unicode as objects, 1 character per, and strings were arrays of theose objects. A convenient OO-schema, but with today's hardware and today's memory limits, totally impractical.

There has to be a better way.

Examine what is said, not who speaks.

"Efficiency is intelligent laziness." -David Dunham
"Think for yourself!" - Abigail
"Memory, processor, disk in that order on the hardware side. Algorithm, algoritm, algorithm on the code side." - tachyon

[reply]