BrowserUk has asked for the wisdom of the Perl Monks concerning the following question:

Devel::Size is a very useful module for working out where your memory is being consumed, but it has one major caveat. In the process of determining the size of complex structures, itself consumes prodigious amounts of memory.

Most of that is consumed tracking references to a) prevent recursing endlessly around circular data structures; b) to avoid counting the same substructures multiple times. As all D::S requires is a boolean answer as to whether a given address has been seen (and counted) before, using a hash is a convenient but costly way to do it.

I've switch that to using a (segmented) bitstring, with the result that the following one-liners require a tiny fraction of additional memory for D::S to do it's thing, relative to the current cpan version, as demonstrated by the following one-liners: 

c:\test>perl -MDevel::Size=total_size -lwe
"$h{ $_ } = [1 .. $_] for 1 .. 1e3;scalar<>;print total_size( \%h ); s
+calar <>;"
11,196k
10115049
46,260k

c:\test>perl -MDevel::Size=total_size -lwe
"$h{ $_ } = [1 .. $_] for 1 .. 1e3;scalar<>;print total_size( \%h ); s
+calar <>;"
11,196k
10115049
11,240k

c:\test>perl -MDevel::Size=total_size -lwe
"$h{ $_ } = [1 .. $_] for 1 .. 5e3;scalar<>;print total_size( \%h ); s
+calar <>;"
253,664k
250591721
1,147,000

c:\test>perl -MDevel::Size=total_size -lwe
"$h{ $_ } = [1 .. $_] for 1 .. 5e3;scalar<>;print total_size( \%h ); s
+calar <>;"
253,644k
250591721
253,776k

[download]

The 3 numbers below each run are: 1) The memory consumed by the process prior to sizing it; 2) the size of that structure as evaluated by D::S; 3) the memory consumed by the process after D::S::total_size() runs. The different between the first and third figures is the overhead of the tracking mechanism.

The first set of figures in each pair are those for the cpan version 0.71; the second for my modified version (dubbed 0.71x for now)

As you can, in these tests, rather than incurring an overhead of ~4 times the requirement of the data-structure being measured, it is just a couple of hundred KB in both cases.

This post is to ask for testers on other (non-win32; 32-bit only currently*) platforms that think they have existing testcases that will exercise this well, to email me at vev6s4702 sneakemail.com (complete) and I'll send them a new distribution (30k) for testing.

(*)I hope to extend that to 64-bit too, but will need the cooperation of someone with a (preferably win64) suitable platform.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."

Replies are listed 'Best First'.
A new (Win32-only) version Devel::Size
by BrowserUk (Patriarch) on Oct 07, 2008 at 09:52 UTC

    If anyone wants a very fast, reliable version of Devel::Size and they are using Win32, please /msg or email me.

    By way of example, the datastructure that this command creates: 

    perl -MDevel::Size=total_size
-wle"$h{ $_ } = [ \( 1 .. $_) ] for 1 .. 3e3;
      scalar<>; print total_size( \%h ); scalar <>"

    [download]

    Takes the standard version of Devel::Size 45 seconds to size, and consumes a whopping 900+MB over and above the memory required by the data structure itself.

    With my modified version, it takes less than 3 seconds and require less that 10MB of additional memory.

    It also fixes many of the existing known traps & bugs.

    If your on another platform sorry. The code would run perfectly well where you are but would need changes to the makefile, but apparently no one knows how to make them!).

    An extended example that sizes the entire opcode tree with a bunch of common large modules loaded and showing the diagnostics produced that would otherwise be segfaults with the standard version:

    Read more... (4 kB)
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
[select]
Re: X-platform testers wanted: new version of Devel::Size
by Burak (Chaplain) on Oct 02, 2008 at 16:52 UTC
    I think that the biggest problem with Devel::Size is the unsupported CODErefs. It even crashes perl when you use them :p And makes this module totally useless IMO (FYI: see the RT queue for the related bug)
[reply]

      I think you should probably just upgrade, because the current cpan version does coderefs just fine: 

      c:\test>perl -MDevel::Size=total_size 
-wle"sub x{ $_[0]*10 } print total_size(\&x)"
2884

c:\test>perl -MDevel::Size=total_size 
-wle"print total_size(\%::)"
199700

      [download]
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
        Unfortunately new version (0.71) does not fix this issue. And in fact, if this was fixed, I think the the related bug was closed too :) Here is a little sample that'll crash your perl: 
        perl -wle "use strict;use Devel::Size qw(total_size);my$t=sub{my$x='';
+};print total_size($t);"

        [download]
        You'll see something like: 
        Free to wrong pool 286ee8 not 286f30.

        [download]
        And perhaps a Windows crash dialogue (if you didn't disable it).
[reply]
[d/l]
[select]
Re: X-platform testers wanted: new version of Devel::Size
by DrHyde (Prior) on Oct 06, 2008 at 11:00 UTC
    Why not just upload a dev release to the CPAN and let the CPAN-testers do it for you?
[reply]
      Why not just upload a dev release to the CPAN and let the CPAN-testers do it for you?
      1. Firstly, it isn't my module so I can't do that.

        I made changes to solve a particular requirement I had, namely sizing data structures close to the limits of my physical address space. They proved so successful that I thought it worth a little effort to try and make them generally available.

      2. Second, I need people, not automated test bots who's purpose in life is generate meaningless statistics.

        The code is XS/C, and the changes I made introduced some particularly difficult elements to port cross platform. Namely unions of pointers and bit-field structs.

        I'm a one-platform/one-compiler developer, and I knew I would need expert assistance to stand a chance of making this work on little-endian/big-endian 32/64-bit etc. Not just those with spare bandwidth and cpu to run automated smoke tests.

      3. I needed people happy to volunteer their time to work with me.

      'nuff said?

      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
        Firstly, it isn't my module so I can't do that.

        I believe you can, it just won't get indexed.

        Second, I need people, not automated test bots who's purpose in life is generate meaningless statistics.

        That isn't their purpose. However, if you display that sort of attitude then the people volunteering their time and resources might ignore you. So I can see why you might have that incorrect belief.

        I needed people happy to volunteer their time to work with me.
        When module authors contact me about test results that I've sent in, I'm happy to work with them as long as they're polite. Other testers try to be as helpful.
[reply]

Back to Seekers of Perl Wisdom