in reply to Hash key or value string reuse?

First, "I'm looking to reduce the overall hash size", my question is why? I've worked with some hashes that are like 200MB is size. How big is yours and why do you think you need to reduce its size?

I will point out that any kind of structure that you'd want, say like a binary-tree or whatever can be implemented in Perl. In no way are you limited to the basic "built-in" data types.

Of course there will be a trade-off between performance and memory size. In general if you have enough memory, don't make that tradeoff! Of course besides reducing speed, this adds complexity which normally is not good unless you really need it!

If you have say a whole bunch of objects that you'd like to store more efficiently, then look at ObjectTemplate.

If you are starting to get "into the guts", then I would recommend "Advanced Perl Programming" by Srinivasan.

Anyway back to original question: the key and value are both stored. There isn't any super tricky thing that "re-uses" strings via a pointer to that string. If that happened, then it would be necessary to keep track of where that string was used before and then to "split it into 2 strings" if you changed it in one place and not in another. So if you use string "bcd" in one hash and "bcd" in another other hash (maybe a subhash or the first hash) as a key, that string "bcd" is replicated.

Anyway this is getting into a complex subject and I go back to my question: why does this matter to you?

Replies are listed 'Best First'.
Re^2: Hash key or value string reuse?
by QM (Parson) on Jun 16, 2009 at 21:11 UTC
    How big is yours and why do you think you need to reduce its size?
    I have a hash on disk that exceeds 42GB. I use a hash to avoid duplicates. Walking the hash through DBM::Deep takes about 5x longer than the equivalent memory hash. Of course, I haven't tried it on one this big, so it's much likely worse.

    There isn't any super tricky thing that "re-uses" strings via a pointer to that string.
    Thanks. I was just wondering if Perl already did this for me. Since that's not the case, it might be worth a try.

    -QM
    --
    Quantum Mechanics: The dreams stuff is made of