in reply to Re^3: Completeness of Devel::Size answers
in thread Completeness of Devel::Size answers

If you have the ability to monitor the process memory usage as the hash is built, you'll see the graph of the memory allocated versus time looks something like...

I've dug a bit deeper. I got the hash building to stop each time the number of buckets increased. This is what I saw:

    : Entries:   Used/Buckts : Expect :  Heap  :  Aux   :
    :--------:-------/-------:--------:--------:--------:
    :       0:      0/     0 :   0.0M :   0.9M :        :
    :      10:      9/    10 :   0.0M :    "   :        :
    :      20:      E/    20 :   0.0M :    "   :        :
    :      40:     1B/    40 :   0.0M :    "   :        :
    :      70:     38/    80 :   0.0M :    "   :        :
    :     130:     67/   100 :   0.0M :    "   :        :
    :     260:     CD/   200 :   0.0M :    "   :        :
    :     550:    1A8/   400 :   0.1M :   1.0M :        :
    :    1100:    368/   800 :   0.1M :   1.2M :        :
    :    2100:    66D/  1000 :   0.2M :   1.3M :        :
    :    4100:    C9B/  2000 :   0.4M :   1.7M :        :
    :    8200:   1913/  4000 :   0.8M :   2.2M :   0.3M :
    :   16400:   327F/  8000 :   1.7M :   3.4M :   0.8M :
    :   32800:   64D5/ 10000 :   3.3M :   5.6M :   1.8M :
    :   65600:   CA94/ 20000 :   6.7M :  10.1M :   3.8M :
    :  131100:  1940F/ 40000 :  13.4M :  19.1M :   7.8M :
    :  262200:  325A8/ 80000 :  26.8M :  37.2M :  15.8M :
    :  524300:  64A98/100000 :  53.5M :  73.3M :  31.8M :
    : 1048600:  C9691/200000 : 107.0M : 145.6M :  63.8M :
    : 2097200: 192DE0/400000 :  -0.0M : 290.0M : 127.8M :
    : 4194400: 326095/800000 :  -0.0M : 578.9M : 255.8M :
    : 5000000: 3977F6/800000 :  -0.0M : 683.7M : 255.8M :
Which shows the effect you described, the hash being rebuilt with more buckets as it grows, and the heap growing with it. (The Used/Buckts is given in hex.)

This table also shows the expected size of the hash, according to Devel::Size::total_size() on a separate run -- where it says -0.0M I don't have a value, because life is too short.

The figures for actual memory use are given by the System Monitor. The Heap is present when Perl starts. The Aux appears as the hash grows -- FWIW, I observe that it's at the far end of VM space, below the Stack.

NB: the actual memory use is taken when my test is run with the Devel::Size::total_size() calls commented OUT -- so the figures are not affected by its overheads.

So, assuming that the hash is created on the heap, on the face of it, either Devel::Size::total_size() is under-reporting, or the construction of the hash is leaving free space in the heap. Anyway, the discrepancy is ~38 bytes per entry or ~19 bytes per bucket. (I assume that Devel::Size::total_size() does take into account the buckets overhead ?)

I tried reducing the key size from 33 to 17. I got the same discrepancy.

The other puzzle is what the Aux is. It appears to be something to do with the rebuilding of the hash. The size is independent of the key size. It's 32 bytes per new number of buckets, or 64 bytes per old number of buckets ! In any case, it's a significant chunk of memory (virtual or otherwise) :-(

Replies are listed 'Best First'.
Re^5: Completeness of Devel::Size answers
by BrowserUk (Patriarch) on Sep 23, 2008 at 18:55 UTC
    (I assume that Devel::Size::total_size() does take into account the buckets overhead ?)

    I can confirm it does. Or rather, it certainly used to back circa v0.58, when I understood it's internals enough that I succeeded in supplying a trivial patch. It seems unlikely that it would have suddenly stopped doing so now, but I pulled v0.71 earlier today and I no longer understand what hell is going on inside there.

    For the rest of your post, I am unqualified to comment as I know little about *nix memory management. I have no idea what Aux is, nor even imagine what it could be. Someone with *nix experience will be needed to interprete your numbers and explain what is going on.

    I do wonder if this is related to zentara's recent discovery that *nix can release some memory back to the OS under some circumstances?


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.