njahnke has asked for the wisdom of the Perl Monks concerning the following question:

I have a long-running Web app running under Perl 5.16.0 that processes hundreds of requests per minute and sadly leaks RAM ... very slowly, maybe 10-20 MB per day. I was able to figure out more major memory leaks in the past using the following code (which is set to run once per minute in the main event loop): 
if (!%first_arena) {
    %first_arena = %{arena_ref_counts()};
    
    my $all = Devel::Gladiator::walk_arena();
    for my $it (@$all) {
        if (ref $it eq 'ARRAY') {
            $arrays{refaddr($it)} = time();
        }
    }
    $all = undef;
} else {
    my %current_arena = %{arena_ref_counts()};
    my %increase;
    for my $type (keys %current_arena) {
        if (!exists $first_arena{$type}) {
            $increase{$type} = $current_arena{$type};
        } elsif ($current_arena{$type} - $first_arena{$type} > 2) {
            $increase{$type} = $current_arena{$type} - $first_arena{$t
+ype};
        }
    }
    if (%increase) {
        print "\nincreases\n";
        for my $type (sort { $increase{$b} <=> $increase{$a} } keys %i
+ncrease) {
            print "$increase{$type}\t$type\n";
        }
        print '%client: '.(scalar keys %client)."\n";
        print '%comet: '.(scalar keys %comet)."\n";
        print '@ready: '.(scalar @ready)."\n";
        print '%async: '.(scalar keys %async)."\n";

        my %leaked_arrays;
        my $all = Devel::Gladiator::walk_arena();
        for my $it (@$all) {
            if (ref $it eq 'ARRAY' and !exists $arrays{refaddr($it)} a
+nd ++$array_count{refaddr($it)}{md5_hex(Dumper $it)} >= 3) {
                $leaked_arrays{refaddr($it)} = $it;
            }
        }
        $all = undef;
        if (%leaked_arrays) {
            my $zero_length = scalar grep { scalar @{$leaked_arrays{$_
+}} == 0 } keys %leaked_arrays;
            my $non_zero_length = scalar grep { scalar @{$leaked_array
+s{$_}} > 0 } keys %leaked_arrays;
            print "leaked arrays: $zero_length zero length, $non_zero_
+length non zero length\n";
        }
    }
}

[download]
After a half hour or so, this prints: 
increases
24202   SCALAR
7482    REF
5966    HASH
5911    REF-HASH
4387    ARRAY
592     REF-SCALAR
552     REF-DBI::st
368     REF-DBI::db
368     DBI::st
184     DBD::Pg::st_mem
155     GLOB
74      CODE
57      REF-IO::Socket::INET
30      IO::File
29      IO::Socket::INET
22      REGEXP
%client: 28
%comet: 28
@ready: 0
%async: 0
leaked arrays: 3830 zero length, 156 non zero length

[download]
The only real leaks are the ARRAYs. Growth of non zero length ARRAYs is almost flat, but zero length ones grow at an average rate of 120 per minute! Why does it think it needs so many zero length ARRAYs?!

Does anyone know what kind of practices could cause this bizarre behavior? Or maybe some other tools I could use to find the source of the leak? A way to tell the line on which the zero-length ARRAYs were allocated would be perfect. I know about Devel::LeakTrace::Fast, but it requires a perl built for debugging, doesn't it? I can do that if I need to, but ...

Thank you, monks!

Replies are listed 'Best First'.
Re: Leaking 0-length ARRAYs!?
by Athanasius (Archbishop) on Jun 30, 2012 at 13:19 UTC
    I know about Devel::LeakTrace::Fast, but it requires a perl built for debugging, doesn't it?

    Test::LeakTrace seems to do what you need, and it does not require a debug build:

    #! perl
use strict;
use warnings;
use Test::LeakTrace;

our $ref;

my @info = leaked_info
{
    my @array = ();
    $ref = \@array;
  # undef $ref;
};

print join(', ', @$_) for @info;

    [download]

    Output (leaked SV, filename, line number):

    ARRAY(0x1ddd33c), 99_SoPW.pl, 10

    [download]

    But, with undef $ref; uncommented, it gives no output (as there is then no leak).

    HTH,

    Athanasius <°(((>< contra mundum

[reply]
[d/l]
[select]
[reply]
Re: Leaking 0-length ARRAYs!?
by flexvault (Monsignor) on Jun 30, 2012 at 14:29 UTC

    njahnke,

    You may know this already, so it may not actually help your situation, but if you execute this untested code: 

      my @array;
  $array[0] = "stuff";
  $array[10_000] = "more-stuff";
  print scalar( @array ),"\n";

    [download]
    The result is 10,001 which is 2 defined array elements and 9,999 undefined elements. If the above variable was a hash you would have 2 defined and existing elements.

    Are you sure your not created large gaps in your arrays that seem to be 'leaking'?

    Good Luck!

    "Well done is better than well said." - Benjamin Franklin

[reply]
[d/l]
Re: Leaking 0-length ARRAYs!?
by bulk88 (Priest) on Jun 30, 2012 at 14:35 UTC
    Toss more {} blocks around your code to create more scopes. An object's DESTROY isn't called when the ref scalar holding it reassigned, but when scope is left. Create an debugging object that prints to console when its DESTROY() and use a step through debugger. I didn't run your code.

    Is your memory leak reproduced in the code you posted or the code you posted just shows how *you* know its a memory leak?
[reply]

Back to Seekers of Perl Wisdom