in reply to memory (leaks and management)

If you're sure that your program never stops growing, then you can modify it so it runs document processing in a tight loop, over the same or different documents, and see if memory usage becomes more aggressive. If yes, you can use Devel::Size if you suspect particular structures, or in general, comment parts of your program until it stops leaking. If we're talking about "finding exactly", this is the only sure method I know, that allows to narrow most of memory leaks ( in finite time :)

I don't think the leaks can be attributed to Perl, unless your program indeed creates circular references, but then it is the program logic that has to be fixed. More probably there's a 3rd party module that is leaking, in which case you'll locate it by reducing your program more and more until all you have is several lines of code, a reproducible showcase. You might want to send it further to developers (and find out that the bug was fixed several versions back, can happen too :) Therefore, before even you begin the hunt, try installing all the latest modules and see if the problem persists.

Good luck!

Replies are listed 'Best First'.
Re^2: memory (leaks and management)
by soliplaya (Beadle) on Mar 28, 2007 at 09:26 UTC
    Thanks. I will have a look at Devel::Size.
    Other than that, not necessarily in order of relevance :
    - Most of the modules are up-to-date, or else modules I have used before without noticing leaks of this magnitude
    - The program is difficult to whittle down, as one step generally depends on the results of the steps before. For instance, the program takes a MS-Office document and converts it to OpenOffice in one step, then in the next step uses the OpenOffice version to extract the text. So I cannot really strip the first part and run the second only.

    I have noticed the following funny thing : at some point, I run an external utility using system(). It can take a while to run, during which my program waits for the return of the system() call. Well, sometimes during that wait, the memory used starts decreasing, as if Perl was using the time to do some cleanup. Is this possible, or am I having hallucinations ?

    A possibly related question : when exactly does Perl decide to embark on a cleanup round ? and is it possible to provoke it ?

[reply]
      I don't think it is Perl, (well yes, perl does call free() once in a while), but rather win32 memory manager that keeps memory for the process for a while.

      As for whittling down, I'd suggest being a bit more creative :) for example, yes indeed, you cannot take MS-to-OO transition out, but you can pre-create the OO documents that correspond to the MS ones, and change the program so that instead of running the actual conversion, it loads the corresponding OO doc from the disk. The same is valid for OO-to-text - comment out the OO part, but save the text separately and use in in the reduced program.

      hope this help, /dk

[reply]

In Section Seekers of Perl Wisdom