If it was me, for that sort of size of data I wouldn't be using any sort of tied hash. Do you need a hash to pass in to someone else's API? (In which case you couldn't assume they wouldn't force all data into ram at once anyway).

What's the scenario that you need to use it in? What format does your data come in? If you need to do fairly sequential access then there are a lot of storage/access options. If you need a lot of random access I'd probably suggest importing the data into an sqlite database (a one-time process) and then refer to that data from your analysis script. Sqlite will automatically do memory based caching etc. for you so it should be surprisingly fast (and free you from worrying about memory utilisation).

In reply to Re: Disk based hash (as opposed to RAM based) by aufflick
in thread Disk based hash (as opposed to RAM based) by techtruth

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.