Another possibility in a different direction is to consider using SQlite. All you have to do install DBD:SQLite, then just "use DBI;". DBI will figure out what to do from the connect statement.

SQLite avoids all the account setup and admin headaches of a traditional SQL server - it stores the data as just a single regular file and their are no "accounts". I've found the performance to be very good and this solution scales easily. One nice feature is that the amount of cache that it uses can be varied dynamically. I run it way up to speed up indexing operations and then run it back down for normal operation.

I don't know enough about your application to say for sure that this is a good idea for you or not. But this has become my "go to" solution for disk resident DB. It supports a big subset of SQL, but you can use it in a simple way without having to become an SQL guru. Maybe you just have a single un-normalized table and index one column as the "key".

Update: this idea would be appropriate if it helped somehow in the processing of this huge hash, if you had to search for stuff that would be part of the "values" to the keys? If the job is just a matter of retrieving the set of data associated with a single key, I would think that browserUk's idea of making the "data" a single string instead of a reference to an array of strings would make a lot of sense. This also reduces memory requirements somewhat as a single string takes less memory than an array of strings.

In reply to Re: Hash of Arrays - Pushing Array references and memory impacts question by Marshall
in thread Hash of Arrays - Pushing Array references and memory impacts question by alittlebitdifferent

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.