Re: understanding devel::leak
by almut (Canon) on Oct 04, 2007 at 10:43 UTC
|
Is there anything missing that compiling with -DDEBUGGING would add?
Yes, if your perl had been built with -DDEBUGGING, Devel::Leak would
call its (then available) routine sv_dump() on the things in
question, which would produce output similar to what you'd get with the
module Devel::Peek. How much that helps, though, will depend
on the particular situation...
Anyhow, most problems with memory leaks in Perl code are related to
circular references. Consider the following
for (1..1000000) {
my $h = {};
$h->{myself} = $h;
}
This harmlessly looking piece of code consumes about 130MB (on my
machine). Reason is that the anonymous hashes (referenced by $h)
are not being freed, because they're holding a reference to themselves,
so their reference count doesn't drop to zero when the $h goes
out of scope...
| [reply] [Watch: Dir/Any] [d/l] [select] |
|
Thanks for clarifying the debugging thing, I had a feeling something must be missing. If anyone has any pointers to getting devel::leak fully working I'd be grateful.
I don't think I've created any circular references, I've certainly not got any re-entrant code. However, I have loads of anonymous arrays. I did find_cycle and Dump() a few variables at function exit points and found all had ref counts greater than 1. I'll see what I can find with devel::peek.
edit Oops, Dump() is devel::peek, I was associating it with data::dumper. Ho hum.
I've just added an undef ....; to the end of all relevent functions to correspond with all the my ......; but I feel like I'm guessing. There must be a better way to get an idea of memory use than watching perfmon.
| [reply] [Watch: Dir/Any] |
|
I've just added an undef ....; to the end of all relevent functions
to correspond with all the my ......;
Whether undef-ing works, depends on what exactly you undef.
In the trivial example above, undef $h at the end of the loop wouldn't help (as that's implicitly
happening anyway), while undef %$h, undef $h->{myself} or delete $h->{myself} would help.
Point is that you have to break the circle before the data structure becomes unreachable...
In real life, self-referencing circles are often not immediately
evident, as they may come into being indirectly through several data
structures, e.g. A references B, which references C, which ...
references A — or some such. Your ref counts > 1 might hint at
such a situation, unless you have another good explanation for them :)
Sometimes (if all else fails), it helps to step by step disable
parts of the program, until the problem goes away. In this case, look
more closely into whatever that last part was, etc. — Without
seeing any actual code, it's hard to come up more specific tips.
| [reply] [Watch: Dir/Any] [d/l] [select] |
|
|
|
Re: understanding devel::leak
by Nova (Novice) on Oct 04, 2007 at 15:25 UTC
|
Hi nonk ,
This theme seems to be quite common , memory gets eaten up and is not released .
The issue is eroding my own confidence in proceeding to develop a perl based tool which I would like to be able to control the release of MEM , when not needed via the best possible code .
On my NT PC / with Perl running under ActiveState , the following code eats up 92,608 KB MEM . After the undef
the MEM usage drops only to 90,572 KB MEM . An awefull lot
of Residual MEM is not returned to the O/S until the program exits .
I know that I am pushing an anonymous array into X , but it really concerns me that undef is not reclaiming the MEM .
Can my code be modified to allow the MEM to be reclaimed to the O/S . I have not found a way
Nova
<code>
#!perl
{
my $i=0 ;
my @X ;
while ( $i < 500000 ) {
push @X , $i , $i , $i , $i ;
if ($i%10000==0) { printf "Index %-7d %-7d \n\n" , $i , $X$i2 ;}
$i++ ;
} ;
printf "Check MEM usage then Return to undef X" ;
<> ;
undef @X ;
};
printf "Enter Return to quit program" ;
<> ;
<\code> | [reply] [Watch: Dir/Any] |
|
The problem you are having is not a 'true' memory leak, i.e. the
memory in fact is being freed. It just isn't being returned to the OS,
but rather to Perl's own memory pool. Those two things are often confused.
The difference is that if you'd run your while loop another
time, it would not allocate another 90MB, but rather reuse the
memory freed upon undef-ing the @X. If it were a memory leak
in the sense that the rest of the thread is talking about, it'd eat up
more and more memory, without you having a chance to reuse it
(without exiting the interpreter).
| [reply] [Watch: Dir/Any] [d/l] [select] |
|
Thank you for your responses and I apologize for the format
problem in my previous post .
From the point of view of the O/S , I can see that , once I have done work via my script which requires MEM , a large chunk of that MEM may no longer be available to the O/S for concurrently running apps .
My perl app is running on WIN32 / ActiveState Perl 5.8.8 . Can I set any variables which would apply the -DPURIFY after
Perl has already been compiled , or the binaries intalled . Further , while the script is running is there a variable which can be examined which I can use to monitor MEM usage .
Kind regards, Nova .
| [reply] [Watch: Dir/Any] |
|
|
|
| [reply] [Watch: Dir/Any] |
|
Thank you Monks for the pointers.
After several weeks of bashing my head against this problem I have at last made a breakthrough, of sorts.
Along my voyage of discovery I've tried:
Devel::leak - no use as I've no idea how to recompile on win32 / activestate. I downloaded the source and read the readme but it just left me even more confused.
Devel::Leak::Object - Don't recall why I gave up on that one.
Devel::Cycle - My data structure is pretty complex so it was entirely plausible that I'd looped somewhere. I thought I wasn't using find_cycle right but it turns out there was nothing there to find.
Devel::Monitor - plain doesn't work on win32 / activestate.
Devel::Peek - I sort of understood what this was telling me but not entirely.
Win32::Perfmon - doesn't work for any of the counters that have instances.
In the end all I could do was step through with the degugger looking for patterns in where perls private bytes increased. Not easy due to the way perl grabs memory in 4k chunks. In the end I got down to one suspect package function. I wrote a fake version of this that returns realistic data. Using this I have proved that there are absolutely no memory leaks in my code. I'd be quite please with this if only it did not render 6 weeks effort pretty worthless.
The module causing the problem is Win32::OLE:OPC and the function is SyncRead.
I'll have a brief look inside the package but I think I reached my limit. It's entirely possible that the problem isn't in that package or even in any of it's perl dependancies but elsewhere in OLE or COM.
Bit depressing really.
nonk
| [reply] [Watch: Dir/Any] |
Re: understanding devel::leak
by brycen (Monk) on Mar 21, 2009 at 00:16 UTC
|
| [reply] [Watch: Dir/Any] |