Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?

Memory leak detection

by runnerup (Novice)
on Sep 07, 2014 at 14:10 UTC ( #1099798=perlquestion: print w/replies, xml ) Need Help??

runnerup has asked for the wisdom of the Perl Monks concerning the following question:

I created a perl socket server using use Danga::Socket::AnyEvent and Danga::Socket::Callback which basically used AnyEvent::HTTP

Anyways the problem is that there is a slow memory leak ,and when using Devel::Gladiator arena_table() function I see:

ARENA COUNTS: 475641 REF 333577 SCALAR 105335 REF-CODE 103141 HASH 101462 REF-HASH 99024 ARRAY 82090 REF-Tie::RefHash::Nestable 74231 REF-ARRAY 69084 REF-Tie::RefHash::Weak 49129 Tie::RefHash::Nestable 39068 REF-Danga::Socket::GCallback 17199 Danga::Socket::GCallback 4827 GLOB 4155 CODE 2062 REF-AnyEvent::Loop::io 1201 REF-SCALAR 1048 IO::File 1032 REF-IO::Socket::INET 1031 AnyEvent::Loop::io 1031 IO::Socket::INET 576 REGEXP 58 Regexp 57 REF-Regexp 7 VSTRING 6 Tie::RefHash::Weak 4 Encode::XS 4 REF-Encode::XS 3 REF-Encode::utf8 2 Encode::utf8 2 FORMAT 2 REF-Config 2 REF-version 2 pseudohash 2 version 1 AnyEvent::CondVar 1 Config 1 Encode::Internal 1 Errno 1 LVALUE 1 POSIX::SigRt 1 Protocol::WebSocket::Frame 1 Protocol::WebSocket::Handshake::Server 1 Protocol::WebSocket::Request 1 Protocol::WebSocket::Response 1 REF-AnyEvent::CondVar 1 REF-Encode::Internal 1 REF-Errno 1 REF-POSIX::SigRt 1 REF-Protocol::WebSocket::Frame 1 REF-Protocol::WebSocket::Handshake::Server 1 REF-Protocol::WebSocket::Request 1 REF-Protocol::WebSocket::Response

The first main problem I think is that REF-Tie::RefHash::Nestable, Tie::RefHash::Nestable seem way too high.

The only use of them is

tie %clients, 'Tie::RefHash::Weak'; tie %users, 'Tie::RefHash::Weak'; tie %connectionCheck, 'Tie::RefHash::Weak'; tie %connectionType, 'Tie::RefHash::Weak'; tie %hs, 'Tie::RefHash::Weak'; tie %frame, 'Tie::RefHash::Weak'; tie %rooms, 'Tie::RefHash::Nestable';

But, when counting/logging them using the following code:

print FILE '%clients: ' . scalar(keys %clients) . ' ' . total_size +(\%clients) . "\n"; print FILE '%users: ' . scalar(keys %users) . ' ' . total_size(\%u +sers) . "\n"; print FILE '%connectionCheck: ' . scalar(keys %connectionCheck) . +' ' . total_size(\%connectionCheck) . "\n"; print FILE '%connectionType: ' . scalar(keys %connectionType) . ' +' . total_size(\%connectionType) . "\n"; print FILE '%hs: ' . scalar(keys %hs) . ' ' . total_size(\%hs) . " +\n"; print FILE '%frame: ' . scalar(keys %frame) . ' ' . total_size(\%f +rame) . "\n"; print FILE '%rooms: ' . scalar(keys %rooms) . ' ' . total_size(\%r +ooms) . "\n";

The output is low such as:

%clients: 1035 2919225 %users: 1038 2919225 %connectionCheck: 767 2919225 %connectionType: 1035 2919225 %hs: 1 2919225 %frame: 1 2919225 %rooms: 213 3292324

All of the references are to RefHash objects are to Danga::Socket::Callback and are deleted by using code such as delete($users{$client}) when the client disconnects. It seems impossible that the scalar(keys %hashes) numbers could add up to the large 69084 REF-Tie::RefHash::Weak number shown on arena counts. Am I missing something? Thanks

Replies are listed 'Best First'.
Re: Memory leak detection
by AppleFritter (Vicar) on Sep 07, 2014 at 15:27 UTC
    Howdy and welcome to the Monastery, runnerup! Can you show us your server's code? It's a bit difficult to diagnose memory leaks without being able to see the leaking program, as I'm sure you'll agree.
      The code itself is over 2000 lines, so I think it would be difficult to post or analyze.
      However, my question is actually how can ARENA COUNTS show
      69084 REF-Tie::RefHash::Weak

      when all the adding all the scalar(keys %HASHS) of all the hashes using Tie::RefHash::Weak, scalar(keys %EACHHASH) added together is much much less than 69084.
      Is there any common case that can happen?
      I am deleting the keys like delete($hash($key)).
      All the keys are to Danga::Socket::GCallback, so there should be no circular references possible.
        However, my question is actually how can ARENA COUNTS show 69084 REF-Tie::RefHash::Weak when all the adding all the scalar(keys %HASHS) of all the hashes using Tie::RefHash::Weak, scalar(keys %EACHHASH) added together is much much less than 69084.

        That is why they call it "memory leak" :) The "arena" goes through internal perl structures to see which structures are allocated. As long as variable is accessible from the user code, it is "good" structure. The problem comes, when the structure is not accessible from user code. This structure is then "leaked". So, if you have a leak, then you won't be able to see leaked structures from your perl code. Of course in this case your count won't match the count of "arena".

Re: Memory leak detection (delete lines until 20 reproduces problem)
by Anonymous Monk on Sep 07, 2014 at 21:15 UTC
      > Reproduce the problem by slowly deleting 1980 lines until you're left with about 20 lines which reproduce the leak :)

      To tell the truth, I can't even reproduce the problem without real users.
      Unfortunately I find it difficult to reproduce problems for socket-server code because it requires multiple users connected.
Re: Memory leak detection
by GrandFather (Saint) on Sep 11, 2014 at 21:01 UTC

    A completely unsatisfactory (in terms of self image anyway) but completely practical solution is the restart the server from time to time. You can even have the server restart itself.

    How much time is it worth spending trying to hunt down this issue? Put a restart procedure in place, possibly triggered by high memory use, then drop the priority for finding a solution down to a level that doesn't get in the way of productive work.

    Perl is the programming world's equivalent of English

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://1099798]
Approved by AppleFritter
Front-paged by Corion
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others rifling through the Monastery: (3)
As of 2023-02-03 22:56 GMT
Find Nodes?
    Voting Booth?
    I prefer not to run the latest version of Perl because:

    Results (30 votes). Check out past polls.