Finding memory leaks

steves has asked for the wisdom of the Perl Monks concerning the following question:

I recently changed about a half dozen base class packages used by most of my code. When I rolled these out I found that my process size (Solaris) was growing at a pretty good clip. When I use the old versions this does not happen.

I can't find this for the life of me. There are two self references, managed via WeakRef. I've check both of these and they are not a problem. I've also used Devel::ObjectTracker and it finds nothing. I boiled my test case down to a loop that just creates a base class instance over and over again. The reference I get with each creation is identical -- I'd expect a new one if memory wasn't being freed. There are a handful of base data structures created once and modified by a base Class attribute package. I've checked those and can find no growth in them (I checked their arrays and hashes for iterative additions that might have been causing this).

Any ideas for other leak checkers? The new code makes heavier use of AUTOLOAD and does some functional goto's within there. In one case it sets a local value as part of that, but I systematically removed each of those and saw the same growth. Are there better tools or methods some monks might suggest using here? Or are there known leak areas in Perl (I'm using 5.6.1) that I'm not aware of?

Comment on Finding memory leaks

Replies are listed 'Best First'.
Re: Finding memory leaks by Ovid (Cardinal) on Apr 24, 2002 at 15:50 UTC
Check out Devel::Leak. That might help you track this problem down. It helps if you have Perl compiled with -DDEBUGGING. I also have to say that, from what I read from your post, you mentioned that you have reduced your test case down to something that doesn't produce memory leaks. In that case, keep adding things back until you get the memory leak. A test case which doesn't reproduce the problem isn't terribly useful. Also, since you're using Solaris, you might find this article interesting. It talks about the large number of "leaks" in 5.6.1 on Solaris - though there is some disagreement over what constitutes a leak. Cheers, Ovid Join the Perlmonks Setiathome Group or just click on the the link and check out our stats.	[reply]
Re: Re: Finding memory leaks by steves (Curate) on Apr 24, 2002 at 17:28 UTC
I was unclear. My small loop does produce the leak, so I've narrowed it down to a handful of packages used under that loop. And I have one include path of older versions of those that doesn't leak. That's what's frustrating: I've basically done a line by line comparison between the two package sets and haven't found anything. What this means, of course, is that it will turn out to be something embarrassingly obvious and stupid. 8-)	[reply]
Re: Finding memory leaks by perrin (Chancellor) on Apr 24, 2002 at 18:12 UTC
There is only one reliable method of looking for leaks: keep pulling stuff out until the leak stops. It's a pain, but it's really the only way. Don't forget that perl allocates memory in chunks at a time, so it may go over the leaking code a few times without growing. This unfortunately means that you can't just go line by line in the debugger and watch for growth. Some leaky code has to be run a few times before it grabs enough memory to show the leak.	[reply]
Re: Finding memory leaks by gav^ (Curate) on Apr 25, 2002 at 02:18 UTC
I've always had the feeling that long running applications are prone to problems like this and try to avoid them. My workaround is to have the script in a cron job which is called every minute. It basically wakes up, checks to see if there are less than a certain amount of itself running, grabs a set of tasks (and marks them as in progress), processes them, logs action, and then marks them as complete. Lately I've been wondering if there are any disadvantages to this method, but I can't seem to think of any (bar the script start up costs). Obviously this isn't applicable for any networking based daemons. gav^	[reply]
Rex2: Finding memory leaks by Sweeper (Pilgrim) on Apr 25, 2002 at 05:40 UTC
One disadvantage could be variable persistence. If you need to remember a variable for the whole duration of the task, you will have to store this variable in a permanent file (with `Storable` or similar module) and retrieve it later. With possible security problems. I think in your case, you are lucky not to need persistence. In your case, a cron job is good enough to prevent memory leaks.	[reply]
Re: Finding memory leaks by steves (Curate) on Apr 28, 2002 at 23:18 UTC
Okay, so I compiled with -DDEBUGGING and successfully used Devel::Leak. And it shows me some leaks. But those leaks are just memory addresses and I can't find any of the addresses it gives me in my code. Is there a way I'm not seeing to easily map addresses back to my code? I've added debug statements at every point I can find where I create anonymous data and so far no matches.	[reply]