Ouch! It's not your data, but your objects that are killing you.

I just loaded 2_000_000 objects ( hash-based, each containing 1 key - scalar 8 bytes. 1 value - string 80 bytes). I put the 2_000_000 object handles into a hash keyed by the id.

Total data 2M * 80 = 152 MB.

Total memory consumed 700MB.

Add a few nested objects in each of those objects and you can see where your memory is going to.

Imagine if perl stored unicode as objects, 1 character per, and strings were arrays of theose objects. A convenient OO-schema, but with today's hardware and today's memory limits, totally impractical.

There has to be a better way.