Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

Re: A more memory efficient storage structure?

by pg (Canon)
on Dec 31, 2002 at 18:16 UTC ( [id://223410]=note: print w/replies, xml ) Need Help??


in reply to A more memory efficient storage structure?

Let's look at this from Perl internal view, to understand why it uses more memory than you expected.

When you insert a key-value pair, Perl would use its hash function to calculate a non-unique internal key (the internal key, which hash really used for indexing). Focus on the word "non-unique", which means there would be other hash elements sharing the same internal key.

Now perl need to allocate memory for this key-value pair. It can just allocate enough memory for this pair, but that would be really slow, as Perl has to allocate/reallocate memory each time when a new key-value pair is inserted. Performance vs. memory...

What Perl really did is to allocate a chunk of memory to store the pairS related to this internal key.

You can imaging that there would be lots of "waste" (in terms of memory, not speed) at the beginning, when there are not many key collisions. But the situation will get progressively improved, as more key collisions happen. When there is a key collision, there is a big chance that perl doesn't need to allocate memory for this internal key, as there is still space left to store pairS for this internal key, so instead of allocating new chunk of memory, it just fills up whatever allocated already (this means the actual usage of allocated memory is getting improved. Of course you would have to allocate another chunk, after the first chunk is used up. On the other hand, having too many key collisions is not a good news to the performance.)

In your case, your data is not big, so we can reasonablly expect there are not many key collisions, and lots of allocated memory are wasted. 16 times is not a surprise, especially when you are using a two-level hash, at the second level, you actually have many hashes. Even worse, if your second level hashes are all those kind of small ones, having 1 or 2 elements, there would be a big waste.

Replies are listed 'Best First'.
Re: Re: A more memory efficient storage structure?
by Elian (Parson) on Dec 31, 2002 at 19:28 UTC
    At some point I need to put together a "How Perl variables use memory" document, I suppose. That might be useful, though less often than one might think.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://223410]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others wandering the Monastery: (5)
As of 2024-03-28 17:25 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found