This is somewhat self-promoting, but as the question is asked with some frequency here and elsewhere, I think this is in order.

Q: How can I find out how much memory my scalar/array/hash/multidimensional data structure uses?
A: Use Devel::Size

Devel::Size is a module designed specifically to figure out how much real memory a variable is using. This counts all the memory, including all of perl's internal bookkeeping and cached memory, not just what you might be able to see from the perl level. (Including things like the memory used for magic, the variable-sized structures that represent scalars of different types, and preallocated extra bits for arrays and hashes)

Here's a quick example of its use, from the docs: 

         use Devel::Size qw(size total_size);

         my $size = size("A string");

         my @foo = (1, 2, 3, 4, 5);
         my $other_size = size(\@foo);

         my $foo = {a => [1, 2, 3],
                 b => {a => [1, 3, 4]}
                };
         my  $total_size = total_size($foo);

[download]
It does follow references when using the total_size function, and does make sure that no structure is counted more than once, so it's safe to use with self-referential data structures.

At the moment its one failing is that it doesn't handle code refs quite completely (it doesn't trace through the scratchpads) and so will complain if you try it on a code ref, or something that has a code ref. It will do its best, though, and count up the memory that it does know how to find. It also won't trace memory that isn't perl's, for example memory that an external library has allocated for itself.

Thanks are due to kudra who both wrote the docs and prodded this module into existence in the first place.

Disclaimer: I wrote the module, so this isn't exactly an unsolicited testimonial :)

Replies are listed 'Best First'.
Re: Finding the size of a variable
by samtregar (Abbot) on Nov 11, 2002 at 19:22 UTC
    Awesome, but... what I'd really like is a general memory usage profiler. Given a Perl program I'd like to be able to examine memory allocation traces using something like dprofpp (mprofpp?).

    I can't even begin to express how useful this would be. I also have absolutely no idea how to build it. But maybe you do? If the answer is yes, but you don't have the time to code it, I would be happy to do the grunt work!

    -sam

[reply]
      Try building with perl's malloc and -DDEBUGGING enabled. Ilya's wedged an amazing amount of tracking data into that memory allocator. Probably not quite enough to do what you want, but it's close. Unfortunately anything you might want to do will definitely require a custom perl, as you'd need to override the allocation and free routines. (This module has the advantage of being able to work standaline, as it doesn't have to intercept anything)
[reply]
        Thanks, I'll try that. While awaiting your reply I had a minor brainstorm on how to build it. I could augment Devel::DProf (and/or my baby, Devel::Profiler) to capture the value of Devel::Size::total_size() at each data point. Use that to develop total memory allocation profiles for each subroutine. This would probably be effective at catching really large allocations, but it will likely miss small ones due to Perl's use of SV arenas, right?

        UPDATE: I'm dumb. I jumped to the wrong conclusion that Devel::Size::total_size() returned the total amount of allocated memory for Perl. RTFM, sam.

        -sam

[reply]
Re: Finding the size of a variable
by grantm (Parson) on Nov 11, 2002 at 20:22 UTC

    On a related note, I saw Doug MacEachern do a very cool demo using B::Size with Apache::Status. This allowed him to browse through the code which had been compiled into a mod_perl instance and (among other things) see what ops a chunk of Perl compiled down to. The fact that it was point and click accessible using a browser meant it had a very low barrier to entry.

[reply]
      I saw him do something similar a few years ago at one of the OSCONs. Definitely really cool, and I looked at his stuff first before I started this module. It didn't, unfortunately, do what I needed at the time, though the optree walking stuff is likely what's needed to finish up most of the lingering problems with CVs.
[reply]

Back to Meditations