It sounds like you might want to use a tied hash, storing most of the data only on disk, most of the time. Another reply mentioned DB_File/DBM_File, and that is basically it. One problem is that tied hashes need to be, basically, flat datastructures. In English, this means that "you can't just throw a HoH at it, and expect it to just work" (sic). You need to turn the subhashes into scalars, and vice versa. Thus, look at MLDBM/Tie::MLDBM, which will take care of this nitty gritty for you.

In case you're really after a way to keep the data in memory (what do you mean, so you don't need persistent data? ;-), the same concept might save you some memory: keep a plain flat hash as your main data storage, of which the values are frozen records, which you must first thaw (to use the Storable terminology) into a plain hash in order to access the deeper underlying data. That's right, you can use Storable for that.

Somehow, I've got the gut feeling that this would imply reinventing MLDBM.

In reply to Re: A memory efficient hash, trading off speed - does it already exist? by bart
in thread A memory efficient hash, trading off speed - does it already exist? by JPaul

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.