You are probably simply running out of RAM. As others have discussed, Perl hashes should usually be well behaved even when they get really huge. However, it is my experience that running out of RAM means that your code slows down exponentially as its need for RAM grows.

One exception is if you can localize RAM access so that items you access at nearby times are also nearby in RAM (that is, clumped together in clusters; that is, only on a few pages of VM not each on a different page of VM). For example, looping over 2-D arrays in FORTRAN or C row-by-row can be much faster than column-by-column.

But Perl makes heavy use of pointers and so really large Perl arrays don't usually lead to very localized memory use even when you stick to one part of the array. And really large Perl hashes jump all over in RAM.

You could try tieing the hash to a DBM file. This will make the initial hash accesses quite a bit slower, but will prevent the exponential slow-down as the hash gets really large and so may well make your code run much faster in this specific case.

                - tye

In reply to Re: The Upper Limit of Perl's Native Data Structures (RAM) by tye
in thread The Upper Limit of Perl's Native Data Structures by arunhorne

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.