I have come seeking wisdom on what is happening to my quite large and quite memory-hungry perl program.
This program works only under Windows, because it uses some external programs that work only there. To describe the program in general terms : it runs as a daemon (Service), and spends it's time scanning a series of directories in search of new documents. Each time it finds one, it grabs it, processes it in series of steps, and delivers the results in a destination directory. Then it loops and does the same again. It is supposed to do that hundreds or thousands of times a day, each loop being one document.
Processing a document involves many steps, calling many CPAN modules and some external programs
I am watching the program run with the Windows Task Manager, and I see it gradually inflate the quantity of memory it is using. It starts at about 20MB, then in irregular little chunks of a few K, it grows over time to 120 MB (it grows a bit almost each time it processes a document - but not always, not systematically, and not by the same amount each time). I will see it gain 5, 10, 15 30 K, then going back down 20 K, the growing again 5 K, etc.. but eventually settling for somewhat more than it had at the start of the loop.
In the end it grows so much that, when I stop the program (nicely), I see that right before it's exit statement, it spends a while more (5 minutes e.g.) growing even more, in chunks of several K at a time, before starting to deflate in bigger chunks and finally exiting after some 5 minutes of cleanup. I suppose the last bit is Perl's final DESTROYs and cleanup before exiting, and there must be a lot of detroying going on.
I have tried to trace where the gradual inflation is happening, by means of carefully inserted "prompt" instructions, which enable me to peek at the Task Manager's indication of memory used at each significant step, before letting the program continue. I found that way that at one point, where the program is parsing for instance a large OpenOffice document, the memory used can increase quite a lot, to decrease quite a lot right after the parsing. But it does not explain everything, and some memory increases (or decreases) seem largely random to me (for example without connection to the size of the document being processed). It even happens at some point, that the program uses say 100 MB of memory, and suddenly decreases to 60 MB, whereafter it starts growing again in small increments. There is nothing in the logic at that point that would explain why it suddenly reclaims 40 MB at that point, rather than at some other moment. I also see that it is not the calling of the external programs that causes the memory increases.
I have also tried to be somewhat "clean" in the amount of dangling or circular references I'm using. But it is a big and complex program, and I am not such a hotshot programmer, so it's quite possible there are some of these things lurking around. I have of course little control over what some CPAN modules might do internally.
I will add that I have carefully read all the scriptures I could find about perl's memory management, GC, reference-counting, circular references etc.., and of course when I talk above of the memory used by my program, what I see in the Task Manager is the total memory used by the process, perl interpreter included. So I generally understand what I am seeing. But what I would expect, is that my program gradually increases the memory it is using, more or less in relation to the largest document it has processed so far, but than at some points stabilises. But that is not what I am seeing : it seems to grow slowly, but persistently and seemingly without real limit.
The point of this long rambling is : how would one go about finding out exactly what is happening, and possibly correct it ?
Thanks in advance for any light
In reply to memory (leaks and management) by soliplaya
| For: | Use: | ||
| & | & | ||
| < | < | ||
| > | > | ||
| [ | [ | ||
| ] | ] |