monkfan has asked for the wisdom of the Perl Monks concerning the following question:

Beloved Monks,

Sometime in a Perl script we created many data structures. It may contain simple one (Array or Hash) or complex (e.g AoAoA or HoHoA etc etc).

My question is, is there a simple way to measure-up the total memory size used up by our script?

I know there is a Devel::Size module. But it only measure the memory size one at a time for one particular data structure. I can find the total size by summing them up.

What I am looking for is one single stroke to measure the total size. Is there one?
I hope I am asking a reasonable question. Thanks so much.

Regards,
Edward
  • Comment on Total Memory Size Used up by a Perl Script

Replies are listed 'Best First'.
Re: Total Memory Size Used up by a Perl Script
by saintmike (Vicar) on Oct 08, 2005 at 08:02 UTC
    Are you saying you want to know the size of the entire process running the script? On Unix-like systems, Proc::ProcessTable provides a nice API for it:
    use Proc::ProcessTable; my $t = Proc::ProcessTable->new(); foreach my $p ( @{$t->table} ) { if($p->pid() == $$) { print "I'm ", $p->size(), " bytes big.\n"; last; } }
      Thanks so much for your answer.

      My intention actually is to measure the total memory size of all data structure used in my script.
      I'm sorry if I wasn't being clear in my OP.

      Then I try this simple script, comparing it with Devel::Size.
      #!/usr/bin/perl -w use strict; use Proc::ProcessTable; use Devel::Size qw(size); my @arr = ('A' .. 'M'); my $devel_size = size(\@arr); print "With DEVEL::SIZE I'm $devel_size bytes big\n"; my $t = Proc::ProcessTable->new(); foreach my $p ( @{$t->table} ) { if($p->pid() == $$) { print "With Proc::ProcessTable I'm ", $p->size(), " bytes big.\n"; last; } }
      It gives:
      With DEVEL::SIZE I'm 104 bytes big With Proc::ProcessTable I'm 5357568 bytes big.
      The difference here is so big. I wonder why? Does Proc::ProcessTable gives the size which I intend to get? If no, is there a way to achieve my original intention?

      Regards,
      Edward

        As you said yourself, Devel::Size only gives you the size of a single data structure, whereas Proc::ProcessTable gives you the size of the whole running process, including perl interpreter, loaded modules etc.

        That being said, 5MB seems kinda big for what you're doing here. Try EvanCarrol's suggestion as well to confirm the result, it should give you the same as Proc::ProcessTable (search.cpan.org seems to be having some problems atm, so I can't download the modules myself to check.)

Re: Total Memory Size Used up by a Perl Script
by BrowserUk (Patriarch) on Oct 08, 2005 at 09:31 UTC

    Try using Devel::Size::total_size on %::;

    perl> use Devel::Size qw[ total_size size ];; perl> print total_size( \%:: );; [warnings omitted] 264621 perl> @a = 'A'..'M';; perl> print total_size( \@a );; 446 perl> print total_size( \%:: );; [warnings omitted] 265437 perl> @b{ 1 .. 100000 } = 1..100000;; perl> print total_size( \%b );; 4813243 perl> print total_size( \%:: );; [warnings omitted] 6803729

    As you can see, the numbers don't exactly add up, but the do give you a pretty good ballpark figure.

    Now, if only I can work out how to supress that dratted warning.

    Update: I found it: $Devel::Size::warn=0;. With that, you can get a rough idea of where the memory your script uses is being used:


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
    "Science is about questioning the status quo. Questioning authority".
    The "good enough" maybe good enough for the now, and perfection maybe unobtainable, but that should not preclude us from striving for perfection, when time, circumstance or desire allow.

      To answer a few questions asked by /msg.

      1. What is Perl>?

        It's a twenty-line script called p1.pl that is effectively similar to doing perl -de1 except is loads a few useful modules and accepts multiple lines of input until one is terminated with ;;

      2. What is the meaning of the double semi-colons?

        See above.

      3. Why the diff is so big between the two examples: total_size( \@a );; and total_size( \%:: );;

        %\:: is (effectively) the symbol table for whole script, including all modules loaded.

        When you do total_size( \%:: );;, it is accumulating the sizes of most* the code space, and all of the data space use by everything that makes up the script, inclduing that space used by all modules that are loaded, and some of the code and data space loaded (into every script) by Perl itself.

        *See the Devel::Size docs where it explains that the numbers for CVs (code structures) are not complete. Don't expect the accumulated sizes to add up exactly. It gives a good idea of where your script is consuming memory, without being exact.

        It will include the space consumed by any variables created by your program (@a in the example) and various other bits and peices including some of the code and data loaded by the Devel::Size module itself, and the various modules ( Carp, Exporter, XSLoader etc. ) that D::S causes to be loaded.

        The long list in the Update above, is the results of dumping the keys and total_sizes of %:: individually from within p1.pl, and shows amongst other things that it also loads Data::Dumper, and Benchmark.

        It also shows various global structures set up by Perl. Eg.

        • %ENV is using 5,557 bytes.
        • %SIG is using 3,705 bytes.
        • %INC is using 2,495 bytes.
        • @ARGV is using 405 bytes.
        • With a little thought it is possible to work out most of the others.For example, the character '↨' using 407 bytes is ascii 23:
          perl> print ord '↨';; 23

          which translates to $^W, and '↑' is 24, $^X which is the name of the script.

        • ...and so on.

      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
      "Science is about questioning the status quo. Questioning authority".
      The "good enough" maybe good enough for the now, and perfection maybe unobtainable, but that should not preclude us from striving for perfection, when time, circumstance or desire allow.
Re: Total Memory Size Used up by a Perl Script
by EvanCarroll (Chaplain) on Oct 08, 2005 at 08:05 UTC
    perl -MPOSIX -e'print `ps --no-header -p @{[getpid()]} -o sz`'


    Update:
    BrowserUk says Re Re: Total Memory Size Used up by a Perl Script Why POSIX/getpid() and not $$?
    Because I'm an idiot. Post an insulting comment so I can upvote you!

    perl -e'print `ps --no-header -p$$ -osz`'


    Evan Carroll
    www.EvanCarroll.com