Leak Hunting

the_slycer has asked for the wisdom of the Perl Monks concerning the following question:

I posted Memory usage & hashes of lists of hashes of lists a couple of days ago regarding memory usage in a foreach loop. Here is some updated code, with some attempts at finding the leak.

I am still unsuccesful, this code uses up a large amount of memory very quickly (~200 meg after ~100 iterations), then slowly releases it, until it's using ~50 meg. Says to me that there is a memory leak somewhere, but here's the code:

my %tick_inf;
foreach (sort keys %tickets)
{
    my $ticket = ((split " ",$_)[0]);
    (%tick_inf = ars_GetEntry($remedy,$schema,$ticket))
        || warn "Could not retrieve $ticket: $ars_errstr";
    ++$count;
    print "$count\n";
    undef (%tick_inf);
    next;
}
[download]

I have attempted a few things to clean this up, most notable is moving the %tick_inf decleration to outside of the foreach loop, and undefing it once I'm done with the data (in this example I'm not doing anything with it). I've also attempted to dump the hashes (using Storable), the results of that show that the serialized hash size averages about 800k. I've used a seperate script to retrieve the hashes and it's mem usage hovers around 9 meg. Some of the values in the hash are AoH's.

This also does not seem to be a platform thing, as I've run the script on Windows 2000, Linux and HP-Unix with the same problem on each. Perl version is 5.6.1 for all of these.

I'm ready to attribute this to a bug with ARSperl so I need to do some hunting in the ARSperl module(s). My questions are:
a) Am I right in assuming this is a memory leak in the ARSperl module? Or can anybody spot anything in this code that would cause the problem that I have described
b) If a is correct then what kinds of things am I looking for in the module - self referencing hashes is one thing that causes memory leaks - what else?
c) Is this even a memory leak - normally those just cause the RAM to be consistently chewed up vs chewed up quickly and released slowly right?

Someone in the last post suggested that I go the political route instead, and indeed in this case that is what has happened - the sysadmins are increasing the user RAM limit on the server. But I am still incredibly curious about this, I don't like to let things go when I don't understand why it is happening.. and I will be using this module in the future (in fact I have a few more times), and I want to make sure that I don't see these issues again.

Comment on Leak Hunting Download Code

Replies are listed 'Best First'.
Re: Leak Hunting by dragonchild (Archbishop) on Oct 01, 2001 at 21:29 UTC
The first thing I would do, now that you have a very simple test case, is to remove the ARSPerl call and replace it with your own call. First off, just return a huge hash. See if that causes the behavior you're seeing. Maybe, return a huge hash of hashes. See if that replicates. Then, keep getting bigger and bigger, deeper and deeper. Try and duplicate the problem and show it's a problem with the loop construct you have right now. If you cannot, your case that it's in ARSPerl is stronger. Then, if you're positive it's in ARSPerl, then contact the people who wrote it and ask if you're using their module(s) wrong. Maybe this is a caveat they should've put in the POD, but didn't. Maybe they have a patch coming out (or even out!) soon. Maybe they would love to have you as a beta-tester for the patch. Note that this doesn't mean you shouldn't look inside ARSPerl. But, a lot of that code may not be in Perl, but in other types of Perl-ish code. (Others could explain it better than I.) ------ We are the carpenters and bricklayers of the Information Age. Don't go borrowing trouble. For programmers, this means Worry only about what you need to implement.	[reply]
Re: Leak Hunting by perrin (Chancellor) on Oct 01, 2001 at 22:54 UTC
First, I don't think you want to set up %tick_inf outside the foreach loop like that. If it's only used within the scope of a single pass through the loop, declare it inside the loop. And I think you want to clear the has by saying `%tick_inf = ()` rather than undef'ing it, but I'm not certain about that. Now, as for finding your leak, you have to do this the old-fashioned way: take stuff out until the problem goes away. Comment out a section of code, run the program, see if it still gets large or not. Keep doing that until you find the part that is making it grow. Then come back and get help fixing that specific piece.	[reply] [d/l]
Re: Leak Hunting - Solved! by the_slycer (Chaplain) on Oct 02, 2001 at 01:31 UTC
It appears as though there is/was a bug in the ARS module. This has been disconvered after some persistence on the ARS_perl mailing list. Apparently nobody had previously noticed that memory was not being freed properly when using the ars_GetEntry call. A possible patch was applied, but it does not seem to fix the trouble. I am waiting for more info from that list. Thanks to everybody here for the help.	[reply]
Re: Leak Hunting by fokat (Deacon) on Oct 02, 2001 at 00:53 UTC
If your memory footprint gets smaller at some point, most likely this is not a memory leak issue. In a leak scenario, the code (in this case, Perl itself or perhaps a module's code) would be loosing track of previously allocated memory areas. Can you provide sample output showing the memory utilisation? You're not looking at the RSS alone, are you?	[reply]