Greetings. I've run into one massive memory leak problem. I have a multithreaded script that goes out and collects info from network devices and dumps it to a file (it will eventually go to mysql). The problem is that after about half an hour the memory usage starts spiraling up out of control. 100 threads start out using about 500 MB but it soon goes to 2 GB. I've paired the script down to the bare minimum trying to locate the leak but no to avail.

A colleage said he heard something about a problem with localized hashes within hashes not freeing memory properly. Anyone heard anything about this? The polling subroutine writes everything to a thread localized hash tree and the main loop writes that hash to a file. The code is like this: 

require Net::SNMP;
use Net::DNS;
require Digest::MD5;
use threads ('stack_size' => 131072);
use threads::shared;
use Thread::Queue;
use Thread::Semaphore;
use Data::Dump qw(dump pp);

{ ## thread entry point
for (;;) {
 local %datahash;
 ..getdevicetopoll..
 &ifpoll($dev);
 print FILE dump %datahash;
}
}

sub ifpoll {
 my %devinfo;
 my %interfaces; ## etc...
 ..snmp a lot of data here..
 %{$datahash{"devinfo"}} = %devinfo;
 %{$datahash{"interfaces"}} = %interfaces;
 return;
}

[download]
All the code is pretty much like that. Collect data in my'd hashes. Then assign that data to the main local'd hash and write out. I checked the symbol table (use Symbol) and %datahash is indeed blank. I even wrote a sub that walked %datahash and manually deleted each key. No difference. I also used the Internals module to check ref counts on the intermediate variables but they're not increasing. My fear here is that there's some stealth "closure" situation going on. Something preventing the lexical data from being garbage collected. Every variable is either my'd or local'd.

Any ideas on what this could be? Any ideas on how to even troubleshoot this? I'm using Solaris 8 and ActiveState Perl 5.8.8 build 817. I've tried build 820 and compiling my own Perl with -DDEBUGGING. No difference.

If anybody has any ideas I'ld be greatly appreciative. Thanks.

In reply to Massive Perl Memory Leak by wagnerc

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.