I have to concur with waswas-fng, your data, or rather your keys must be such that your approaching worst case behaviour. I tried your original code on my 233Mhz P2 and loading the hash took a little under 8 seconds.

To acheive the load times of 30 minutes you are quoting. you are either

  1. extremely unlucky.
  2. using an Apple II or ZX 81 :)
  3. your data has been picked to deliberately induce the pathological behaviour.

If a) is the case, my commiserations, but I do have a work-around for you. If you are currently using 5.8, revert to 5.6.1. If your currently using 5.6.1 or earlier, switch to 5.8. The hashing function changed between these two builds and if your data is genuinely inducing the O(n^2) insertion behaviour on one build, it will perform as you would hope on the other.

Examine what is said, not who speaks.
"Efficiency is intelligent laziness." -David Dunham
"When I'm working on a problem, I never think about beauty. I think only how to solve the problem. But when I have finished, if the solution is not beautiful, I know it is wrong." -Richard Buckminster Fuller


In reply to Re: Slurping BIG files into Hashes by BrowserUk
in thread Slurping BIG files into Hashes by Elgon

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.