Memory leak detection

runnerup has asked for the wisdom of the Perl Monks concerning the following question:

I created a perl socket server using use Danga::Socket::AnyEvent and Danga::Socket::Callback which basically used AnyEvent::HTTP

Anyways the problem is that there is a slow memory leak ,and when using Devel::Gladiator arena_table() function I see:

ARENA COUNTS:
 475641 REF
 333577 SCALAR
 105335 REF-CODE
 103141 HASH
 101462 REF-HASH
 99024 ARRAY
 82090 REF-Tie::RefHash::Nestable
 74231 REF-ARRAY
 69084 REF-Tie::RefHash::Weak
 49129 Tie::RefHash::Nestable
 39068 REF-Danga::Socket::GCallback
 17199 Danga::Socket::GCallback
 4827 GLOB
 4155 CODE
 2062 REF-AnyEvent::Loop::io
 1201 REF-SCALAR
 1048 IO::File
 1032 REF-IO::Socket::INET
 1031 AnyEvent::Loop::io
 1031 IO::Socket::INET
  576 REGEXP
   58 Regexp
   57 REF-Regexp
    7 VSTRING
    6 Tie::RefHash::Weak
    4 Encode::XS
    4 REF-Encode::XS
    3 REF-Encode::utf8
    2 Encode::utf8
    2 FORMAT
    2 REF-Config
    2 REF-version
    2 pseudohash
    2 version
    1 AnyEvent::CondVar
    1 Config
    1 Encode::Internal
    1 Errno
    1 LVALUE
    1 POSIX::SigRt
    1 Protocol::WebSocket::Frame
    1 Protocol::WebSocket::Handshake::Server
    1 Protocol::WebSocket::Request
    1 Protocol::WebSocket::Response
    1 REF-AnyEvent::CondVar
    1 REF-Encode::Internal
    1 REF-Errno
    1 REF-POSIX::SigRt
    1 REF-Protocol::WebSocket::Frame
    1 REF-Protocol::WebSocket::Handshake::Server
    1 REF-Protocol::WebSocket::Request
    1 REF-Protocol::WebSocket::Response
[download]

The first main problem I think is that REF-Tie::RefHash::Nestable, Tie::RefHash::Nestable seem way too high.

The only use of them is

tie %clients, 'Tie::RefHash::Weak';
tie %users, 'Tie::RefHash::Weak';
tie %connectionCheck, 'Tie::RefHash::Weak';
tie %connectionType, 'Tie::RefHash::Weak';
tie %hs, 'Tie::RefHash::Weak';
tie %frame, 'Tie::RefHash::Weak';
tie %rooms, 'Tie::RefHash::Nestable';
[download]

But, when counting/logging them using the following code:

    print FILE '%clients: ' . scalar(keys %clients) . ' ' . total_size
+(\%clients) . "\n";
    print FILE '%users: ' . scalar(keys %users) . ' ' . total_size(\%u
+sers) . "\n";
    print FILE '%connectionCheck: ' . scalar(keys %connectionCheck) . 
+' ' . total_size(\%connectionCheck) . "\n";
    print FILE '%connectionType: ' . scalar(keys %connectionType) . ' 
+' . total_size(\%connectionType) . "\n";
    print FILE '%hs: ' . scalar(keys %hs) . ' ' . total_size(\%hs) . "
+\n";
    print FILE '%frame: ' . scalar(keys %frame) . ' ' . total_size(\%f
+rame) . "\n";
    print FILE '%rooms: ' . scalar(keys %rooms) . ' ' . total_size(\%r
+ooms) . "\n";
[download]

The output is low such as:

%clients: 1035 2919225
%users: 1038 2919225
%connectionCheck: 767 2919225
%connectionType: 1035 2919225
%hs: 1 2919225
%frame: 1 2919225
%rooms: 213 3292324
[download]

All of the references are to RefHash objects are to Danga::Socket::Callback and are deleted by using code such as delete($users{$client}) when the client disconnects. It seems impossible that the scalar(keys %hashes) numbers could add up to the large 69084 REF-Tie::RefHash::Weak number shown on arena counts. Am I missing something? Thanks

Comment on Memory leak detection Select or Download Code

Replies are listed 'Best First'.
Re: Memory leak detection by AppleFritter (Vicar) on Sep 07, 2014 at 15:27 UTC
Howdy and welcome to the Monastery, runnerup! Can you show us your server's code? It's a bit difficult to diagnose memory leaks without being able to see the leaking program, as I'm sure you'll agree.	[reply]
Re^2: Memory leak detection by runnerup (Novice) on Sep 07, 2014 at 15:55 UTC
The code itself is over 2000 lines, so I think it would be difficult to post or analyze. However, my question is actually how can ARENA COUNTS show 69084 REF-Tie::RefHash::Weak when all the adding all the scalar(keys %HASHS) of all the hashes using Tie::RefHash::Weak, scalar(keys %EACHHASH) added together is much much less than 69084. Is there any common case that can happen? I am deleting the keys like delete($hash($key)). All the keys are to Danga::Socket::GCallback, so there should be no circular references possible.	[reply]
Re^3: Memory leak detection by andal (Hermit) on Sep 08, 2014 at 06:48 UTC
However, my question is actually how can ARENA COUNTS show 69084 REF-Tie::RefHash::Weak when all the adding all the scalar(keys %HASHS) of all the hashes using Tie::RefHash::Weak, scalar(keys %EACHHASH) added together is much much less than 69084. That is why they call it "memory leak" :) The "arena" goes through internal perl structures to see which structures are allocated. As long as variable is accessible from the user code, it is "good" structure. The problem comes, when the structure is not accessible from user code. This structure is then "leaked". So, if you have a leak, then you won't be able to see leaked structures from your perl code. Of course in this case your count won't match the count of "arena".	[reply]
Re^4: Memory leak detection by runnerup (Novice) on Sep 11, 2014 at 02:46 UTC
Re^5: Memory leak detection by andal (Hermit) on Sep 11, 2014 at 07:08 UTC
Some notes below your chosen depth have not been shown here
Re: Memory leak detection (delete lines until 20 reproduces problem) by Anonymous Monk on Sep 07, 2014 at 21:15 UTC
...there is a slow memory leak ... The first main problem I think is ... It seems impossible that ... Am I missing something? code itself is over 2000 lines Reproduce the problem by slowly deleting 1980 lines until you're left with about 20 lines which reproduce the leak :) Links on this type of stuff :) Tutorials: Variable Scoping in Perl: the basics, Coping with Scoping , Mini-Tutorial: Perl's Memory Management, Lexical scoping like a fox, Read this if you want to cut your development time in half!, Closure on Closures , perlref#Circular References, Memory leaks and circular references, Circular references and Garbage collection., make perl release memory, about memory management , Re^3: Scope of lexical variables in the main script , Re: What's the weaken here for?, Devel::Cycle, Devel::NYTProf, Devel::Leak Devel::LeakTrace, WeakRef	[reply]
Re^2: Memory leak detection (delete lines until 20 reproduces problem) by runnerup (Novice) on Sep 11, 2014 at 02:43 UTC
> Reproduce the problem by slowly deleting 1980 lines until you're left with about 20 lines which reproduce the leak :) To tell the truth, I can't even reproduce the problem without real users. Unfortunately I find it difficult to reproduce problems for socket-server code because it requires multiple users connected.	[reply]
Re: Memory leak detection by GrandFather (Saint) on Sep 11, 2014 at 21:01 UTC
A completely unsatisfactory (in terms of self image anyway) but completely practical solution is the restart the server from time to time. You can even have the server restart itself. How much time is it worth spending trying to hunt down this issue? Put a restart procedure in place, possibly triggered by high memory use, then drop the priority for finding a solution down to a level that doesn't get in the way of productive work. Perl is the programming world's equivalent of English	[reply]


Perl Monk, Perl Meditation
	PerlMonks