robins has asked for the wisdom of the Perl Monks concerning the following question:

I'm trying to profile my application's memory usage, but it's not working. Basically, I just want to list all variables, sorted on which uses the most memory. This is the current function I've developed, but it doesn't work. It hangs, and I cannot quite understand why. 
sub profile_memory {

    use Devel::Gladiator ();
    use Devel::Size ();
    use Data::Dumper qw(Dumper);
    use Scalar::Util ();

    open(my $fh,">>/tmp/memory_profile_$$.log");

    my $all=Devel::Gladiator::walk_arena();
    my $size_hash={};
    foreach my $sv ( @$all ) {
        next if Scalar::Util::refaddr($sv) == Scalar::Util::refaddr($a
+ll);
        next if Scalar::Util::refaddr($sv) == Scalar::Util::refaddr($s
+ize_hash);
        next if Scalar::Util::refaddr($sv) == Scalar::Util::refaddr($f
+h);
        next if ref($sv) eq 'GLOB'; # Devel::Size segfaults on this
        my $size=0;
        eval {
            #warn(ref($sv)) unless ref($sv) eq 'SCALAR';
            $size=Devel::Size::size($sv);
            #$size=Scalar::Util::refaddr($sv);
        };
        next if $@;
        my $sv_list=$size_hash->{$size} || [];
        next if Scalar::Util::refaddr($sv) == Scalar::Util::refaddr($s
+v_list);
        push @$sv_list, $sv;
        $size_hash->{$size}=$sv_list;
    }

    print $fh ( "-" x 80 ) . "\n";
#    print $fh Dumper($size_hash);
    foreach my $size ( sort { $b <=> $a } keys %$size_hash ) {
    foreach my $size ( keys %$size_hash ) {
        print $fh $size . '=[';
        foreach my $sv ( @{ $size_hash->{$size} } ) {
            print $fh ref($sv) . ", ";
        }
        print $fh ']' . "\n";
    }

    close($fh);

    return 1;
}

[download]

Replies are listed 'Best First'.
Re: Trying to profile my apps memory usage
by BrowserUk (Patriarch) on Nov 18, 2008 at 10:27 UTC
    It hangs,

    That's scant information upon which to make a diagnosis.

    • Does cpu activity cease?
    • Is memory growth stagnant or climbing?
    • Does walk_arena() return?
    • Is your loop still iterating?
    • Where is it hanging and what type of sv was the last you processed?

    You could try this version of Devel::Size. It uses far less memory and has much better trap handling than the official version. It'll probably allow you to do away with this: 

      next if ref($sv) eq 'GLOB'; # Devel::Size segfaults on this

    [download]
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
[select]

      CPU activity is max (99%).

      Memory is stable (not climbing).

      walk_arena() finished.

      It's the printout loop that seems to hang. In fact it seems like it's the foreach (even without the sort) that hangs. The odd this is that if I try to use Data::Dumper to dump $size_hash it hangs too.

      Cannot determine it because the foreach seems to hang.

      I re-ran the tests with v0.72 of Devel::Size. Same result.

[reply]
        Memory is stable (not climbing).

        How much is it using at that point?

        Best guess on the info so far is that you are running out of (physical) memory, and moving into swapping.

        One assumes that the reason you are profiling memory is because you are using a lot of it--prior to deciding to profile. If so, building a hash that contains a reference to every SV in your application is likely to at least quadruple the ammount of memory used. Then creating a list of all the keys of that hash: 

        foreach my $size ( keys %$size_hash ) {
##.................^^^^^^^^^^^^^^^^

        [download]

        is going to stretch that by (guess!) half as much again.

        If you use Data::Dumper on that same hash the memory requirement will likely quadruple again as it uses a hash internally (of the SV addresses), to detect circular and duplicate references.

        Your best bet, (based upon my wild guesswork above), would be to avoid building lists by iterating the hash using while each, and iterating the arrays using the range iterator (.. which doesn't build a list): 

            while( my( $size, $ref ) = each %{ $size_hash } ) {
        print $fh $size . '=[';
        foreach my $i ( 0 .. $#{ $ref } ) {
            print $fh ref( $ref->[ $i ] ) . ", ";
        }
        print $fh ']' . "\n";
    }

        [download]

        That might just allow you to iterate through without breaking the memory bank and moving into swapping.

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
[select]

Back to Seekers of Perl Wisdom