Re^3: Hash memory leak: posible scope issue?

Should I separate them?

No. My thought was that the only way I could see of explaining your results is that your initial declaration:

our %RCache = (); keys( %RCache ) = 1020;
[download]

was in a different package to your reset subroutine:

sub CheckMemoryUsage
{  our %RCache;
...
[download]

and that because you redeclared the hash locally to the reset sub, you were manipulating two different hashes. But that doesn't make sense as the ENTER size would then be zero.

I've been unable to reproduce your results in a simplified test:

#! perl -slw
use strict;
use Devel::Size qw[ total_size ];

our %cache; keys %cache = 1020;
sub resetCache {
    our %cache;
    printf "Before: keys: %u size: %u\n",
        scalar keys %cache,
        total_size( \%cache );
    %cache = ();
    printf "After: keys: %u size: %u\n",
        scalar keys %cache,
        total_size( \%cache );
}

while( 1 ) {
    $cache{ int( rand 2**32 ) } = chr(0) x 2048;
    if( keys %cache >8192 ) {
        resetCache();
    }
}
__END__
C:\test>junk29
Before: keys: 8193 size: 17727631
After: keys: 0 size: 131144
Before: keys: 8193 size: 17727579
After: keys: 0 size: 131144
Before: keys: 8193 size: 17727590
After: keys: 0 size: 131144
Before: keys: 8193 size: 17727614
After: keys: 0 size: 131144
Before: keys: 8193 size: 17727556
After: keys: 0 size: 131144
Before: keys: 8193 size: 17727578
After: keys: 0 size: 131144
Before: keys: 8193 size: 17727621
After: keys: 0 size: 131144
...
[download]

And the question was me speculating about possible scenarios.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

The start of some sanity?

Comment on Re^3: Hash memory leak: posible scope issue? Select or Download Code

Replies are listed 'Best First'.
Re^4: Hash memory leak: posible scope issue? by flexvault (Monsignor) on Jan 03, 2012 at 20:31 UTC
I did the same thing by copying the script and removing all that was not related to %RCache. I could run it 25 times more than the calls in the script and 'no memory leak'. It seems that something I'm doing is making %RCache the victim of another part of the script. I did remove all I/O. In the actual code, the first time I go to the resetCache subroutine is more the 3 minutes into the run, while I could run the reduced script in less than 2 seconds. So there's a lot going on, but it has to be my code. I'm using 'sysseek/sysread/syswrite' because of having boundary problems with 'seek/read/print' or mainly 'print', but I have been pleased with the performance of the 'sys' functions. I'll try to remove less code and see what happens. Thank you for thinking of the packages. I'll keep you informed. "Well done is better than well said." - Benjamin Franklin*	[reply]
Re^5: Hash memory leak: posible scope issue? by BrowserUk (Patriarch) on Jan 03, 2012 at 20:35 UTC
Another thought. You're not running this in a mod_perl or other 'persistant perl' environment are you? With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. The start of some sanity?	[reply]
Re^6: Hash memory leak: posible scope issue? by flexvault (Monsignor) on Jan 03, 2012 at 20:50 UTC
From the command line: 'perl pyrtestdb.plx' I put the test case and module in scope by: `{ my %RCache = (); { test case calling module } { module } }` [download] Until this, is was almost too good to be true, then reality! Thank you "Well done is better than well said." - Benjamin Franklin	[reply] [d/l]
Re^6: Hash memory leak: posible scope issue? by flexvault (Monsignor) on Jan 07, 2012 at 19:07 UTC
BrowserUk, I found the problem(s). I believe I had 3 things wrong and they were all part of the code I posted! I did this by "reverse engineering" the problem. After showing you how the test case and module were put together, I decided to take them apart and load the module with "use". I then decided to use '%RCache' hash to point to the data contained in an array. Now I could re-initialize '%RCache' but the array couldn't be re-initialized. So I thought it's something wrong with the data I'm producing in the index file. And that enabled me to see the actual problem. In the code, I copied the temp buffer to the reference passed from the caller: `substr($$buffer,0,$reclen+4,$tmpbuf); ##` [download] The first thing I did wrong, was that I initialized '$buffer' only once. So when the above was executed and '$$buffer' was larger than '$tmpbuf', it would grow. The 'sub GetSubBuffer' gets called 24,000,000 in this test case, and the data in '%RCache' just kept getting bigger. When I replaced it with: `$$buffer = substr($tmpbuf,0,$reclen+4);` [download] the problem with re-initialize just went away and the test throughput improved from 15m37.516s to 11m16.684s (approximately 30+%). I still don't know why I couldn't re-initialize '%RCache', but it works now. Maybe the failing code created a self-reference. You would know better than I. The third problem was scoping in nature, and had to be fixed just to get the test case to compile after splitting the code. If you notice the '##' after the code, that should have told me to look at that line first, since I add that to code that I think should have additional testing. It worked for the program, since '$reclen' is the size of the actual data, so it's always using the beginning (front) of '$buffer' and when it goes to disk, I'm using 'syswrite' with a count, so the problem was only for the cached data that was growing. Thank you for your help. "Well done is better than well said." - Benjamin Franklin	[reply] [d/l] [select]