in reply to Re: System out of memory while storing huge amount of data
in thread System out of memory while storing huge amount of data

I ran your test on my 32 bit machine. The AoH took 411 MB, then I tried an AoA and that wasn't much better, 336 MB. I conclude that the hash overhead in terms of storage isn't as much as one might imagine! Then I ran the string version and as you observe 52 MB - way smaller than either of the above.

I attribute the difference in MB numbers to greater size of pointers on your 64 bit machine, not to any difference in the benchmark.

  • Comment on Re^2: System out of memory while storing huge amount of data

Replies are listed 'Best First'.
Re^3: System out of memory while storing huge amount of data
by BrowserUk (Patriarch) on Oct 28, 2010 at 15:39 UTC
    I attribute the difference in MB numbers to greater size of pointers on your 64 bit machine,

    Indeed. 64-bit pointers cost heavily.

    Especially as, (today and for the immediate future), on any machine less than something like $250k, the top 24-bits and usually more will be zeros. Even worse when you consider that the bottom 4-bits will also be 0.

    In an XS module I'm writing, I'm experimenting with storing 64-bit pointers in 32-bit fields. By right-shifting 4 places, I get 32-bit values than can cover 64GB, which is as much memory as I'm going to have in the next say 10 years or so.

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

In Section Seekers of Perl Wisdom