Re^4: Hash memory leak: posible scope issue?

I did the same thing by copying the script and removing all that was not related to %RCache. I could run it 25 times more than the calls in the script and 'no memory leak'. It seems that something I'm doing is making %RCache the victim of another part of the script.

I did remove all I/O. In the actual code, the first time I go to the resetCache subroutine is more the 3 minutes into the run, while I could run the reduced script in less than 2 seconds. So there's a lot going on, but it has to be my code.

I'm using 'sysseek/sysread/syswrite' because of having boundary problems with 'seek/read/print' or mainly 'print', but I have been pleased with the performance of the 'sys*' functions. I'll try to remove less code and see what happens.

Thank you for thinking of the packages. I'll keep you informed.

"Well done is better than well said." - Benjamin Franklin

Comment on Re^4: Hash memory leak: posible scope issue?

Replies are listed 'Best First'.
Re^5: Hash memory leak: posible scope issue? by BrowserUk (Patriarch) on Jan 03, 2012 at 20:35 UTC
Another thought. You're not running this in a mod_perl or other 'persistant perl' environment are you? With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. The start of some sanity?	[reply]
Re^6: Hash memory leak: posible scope issue? by flexvault (Monsignor) on Jan 03, 2012 at 20:50 UTC
From the command line: 'perl pyrtestdb.plx' I put the test case and module in scope by: `{ my %RCache = (); { test case calling module } { module } }` [download] Until this, is was almost too good to be true, then reality! Thank you "Well done is better than well said." - Benjamin Franklin	[reply] [d/l]
Re^6: Hash memory leak: posible scope issue? by flexvault (Monsignor) on Jan 07, 2012 at 19:07 UTC
BrowserUk, I found the problem(s). I believe I had 3 things wrong and they were all part of the code I posted! I did this by "reverse engineering" the problem. After showing you how the test case and module were put together, I decided to take them apart and load the module with "use". I then decided to use '%RCache' hash to point to the data contained in an array. Now I could re-initialize '%RCache' but the array couldn't be re-initialized. So I thought it's something wrong with the data I'm producing in the index file. And that enabled me to see the actual problem. In the code, I copied the temp buffer to the reference passed from the caller: `substr($$buffer,0,$reclen+4,$tmpbuf); ##` [download] The first thing I did wrong, was that I initialized '$buffer' only once. So when the above was executed and '$$buffer' was larger than '$tmpbuf', it would grow. The 'sub GetSubBuffer' gets called 24,000,000 in this test case, and the data in '%RCache' just kept getting bigger. When I replaced it with: `$$buffer = substr($tmpbuf,0,$reclen+4);` [download] the problem with re-initialize just went away and the test throughput improved from 15m37.516s to 11m16.684s (approximately 30+%). I still don't know why I couldn't re-initialize '%RCache', but it works now. Maybe the failing code created a self-reference. You would know better than I. The third problem was scoping in nature, and had to be fixed just to get the test case to compile after splitting the code. If you notice the '##' after the code, that should have told me to look at that line first, since I add that to code that I think should have additional testing. It worked for the program, since '$reclen' is the size of the actual data, so it's always using the beginning (front) of '$buffer' and when it goes to disk, I'm using 'syswrite' with a count, so the problem was only for the cached data that was growing. Thank you for your help. "Well done is better than well said." - Benjamin Franklin	[reply] [d/l] [select]