memory sieve discovery

Pstack has asked for the wisdom of the Perl Monks concerning the following question:

Hi to the hallowed!

I have had occasion to use Devel::LeakTrace::Fast in the head script of a suite and carped some interesting stats. Well, what to make of the stats is my current query.

Over 65,000 "leaks" are reported (the log is 5mb). Several thousand are from my own code, and there are some from installed modules also. A few questions to get me started occur straight away:

1. Are these "leaks" lost to the host system (reducing available memory until rebooted), or are they internal to perl in the sense they will be recovered by the host system when perl exits?

2. What significance should I otherwise attach to the stats. particularly the quantity involved.

3. Are the authors of leaky modules likely to be interested in hearing from me ..(over 5000 are recorded against Exporter.pm)..?

For some background, I am attempting to track down cumulative system degradation. KDE slows excruciatingly once the "free memory" reported by KSysGuard goes under 20mb, and I have noticed that then the "cache memory" and "buffer memory" reported has usually grown to a total over 300mb. This may not be anything to do with Perl, of course, but thought it a good place to start looking for eliminations!

I remain your wisdom sponge in these matters.

Pstack

Comment on memory sieve discovery

Replies are listed 'Best First'.
Re: memory sieve discovery by hipowls (Curate) on Jan 18, 2008 at 22:49 UTC
The memory is allocated by the system to the running perl process, when it exits the memory is reclaimed. I'd be inclined to fix the leaks, you may find that you don't have many holes, they just occur in loops. It depends on how much you think the effort is worth. Most module authors welcome bug reports, don't get discouraged by the few who don't;) See here for an explanation of cache and buffer memory.	[reply]
Re^2: memory sieve discovery by Pstack (Scribe) on Jan 18, 2008 at 23:32 UTC
>> .. you may find that you don't have many holes .. I wish. Given a hash-key per unique hole summarises to 500 holes, and 80 of them mine! But cheers, the main concern is hugely reduced.	[reply]
Re: memory sieve discovery by Anonymous Monk on Jan 19, 2008 at 02:31 UTC
Do you have examples of some of these "leaks"? If they are in Exporter, is perhaps assigning to a glob a leak?	[reply]
Re^2: memory sieve discovery by Pstack (Scribe) on Jan 19, 2008 at 05:08 UTC
Here are some others: 651 /usr/lib/perl5/5.8.7/File/Path.pm line 121 582 /usr/lib/perl5/5.8.7/Math/BigInt.pm line 143 2947 /usr/lib/perl5/5.8.7/Math/BigInt.pm line 2303 1834 /usr/lib/perl5/5.8.7/Math/BigInt/Calc.pm line 1420 1472 /usr/lib/perl5/site_perl/5.8.7/HTML/TableExtract.pm line 1020 747 /usr/lib/perl5/site_perl/5.8.7/HTTP/Message.pm line 9 849 /usr/lib/perl5/site_perl/5.8.7/LWP.pm line 10 709 /usr/lib/perl5/site_perl/5.8.7/LWP/Protocol/http.pm line 416 1210 /usr/lib/perl5/site_perl/5.8.7/OLE/Storage_Lite.pm line 11 1194 /usr/lib/perl5/site_perl/5.8.7/OLE/Storage_Lite.pm line 712 2202 /usr/lib/perl5/site_perl/5.8.7/Regexp/Common.pm line 36 3924 /usr/lib/perl5/site_perl/5.8.7/Spreadsheet/ParseExcel.pm line 1929 1793 /usr/lib/perl5/site_perl/5.8.7/Spreadsheet/ParseExcel/Utility.pm line 11 1433 /usr/lib/perl5/site_perl/5.8.7/Spreadsheet/WriteExcel/Formula.pm line 25 2465 /usr/lib/perl5/site_perl/5.8.7/Spreadsheet/WriteExcel/Workbook.pm line 25 1311 /usr/lib/perl5/site_perl/5.8.7/URI/http.pm line 6 1012 /usr/lib/perl5/site_perl/5.8.7/i686-linux/BerkeleyDB.pm line 1441 551 /usr/lib/perl5/site_perl/5.8.7/i686-linux/Compress/Zlib.pm line 16 253 /usr/lib/perl5/site_perl/5.8.7/i686-linux/HTML/Entities.pm line 150 523 /usr/lib/perl5/site_perl/5.8.7/i686-linux/HTML/HeadParser.pm line 74 The leading number in this sample is just the leak count within 1 run of the application -- a batch process. And as I say, I dont know what, if any, significance to attach. Perhaps Exporter is excused, for instance, if globalising is considered "leaky" by definition. And so on. Pstack	[reply]
Re^3: memory sieve discovery by Corion (Patriarch) on Jan 19, 2008 at 13:22 UTC
I think at least part of this report is exaggerated. Devel::LeakTrace::Fast says that most of its reporting starts at `INIT` time to circumvent problems with symbol table aliasing. But the "early" reports for example for HTTP::Message (line 9) come because HTTP::Message uses `require` instead of `use` to load `Carp.pm`. My guess is that all modules by Gisle Aas , most notably all modules comprising LWP, will follow the same allocation pattern (that is, URI::http, LWP). Most likely, others will also follow this pattern. I think to separate the wheat from the chaff of the leaks reported by Devel::LeakTrace::Fast, you will have to write code that runs a small loop exercising exactly the code where the leak is reported, and then watch that program to see whether it grows beyond bounds or not. As another example, line 25 of SpreadSheet::WriteExcel::Formula seems to be the `@ISA` assignment - while that's using up memory, it's not generally considered a leak because that memory is still reachable.	[reply] [d/l] [select]
Re^4: memory sieve discovery by Pstack (Scribe) on Jan 19, 2008 at 21:05 UTC
Re^3: memory sieve discovery by grinder (Bishop) on Jan 19, 2008 at 12:10 UTC
File::Path is pure-Perl, no XS components, so I'm hard pressed to explain what its leak might be. You could install the latest version and if the problem persists, file a bug report. I have a feeling, however, that your problem is more likely to be a coding style that happens to prevent temporaries from being freed at an appropriately early moment. • another intruder with the mooring in the heart of the Perl	[reply]
Re^4: memory sieve discovery by Pstack (Scribe) on Jan 19, 2008 at 20:39 UTC
Re^3: memory sieve discovery by syphilis (Archbishop) on Jan 19, 2008 at 10:56 UTC
I wouldn't give any credence to a report that says that Math::BigInt leaks memory. Cheers, Rob	[reply]
Re^4: memory sieve discovery by Pstack (Scribe) on Jan 19, 2008 at 20:20 UTC