in reply to Memory Leak Package

Reading between the lines of your last couple of OPs, you seem to be parsing multiple XML files with XML::Simple; and doing so in threads in the hope of preventing leaks?

From a couple of quick experiments, it seems that whichever underlying parser -- XML::SAX, XML::SAX::PurePerl, XML::Parser -- you use, they all leak substantial amounts of memory. XML::SAX more so that XML::Parser, but still in the range of 10 to 30MB per iteration for a 12MB xml file.

My suggestion to cure that -- essentially what Zentara suggested earlier, but with a twist -- is to run the parser in a separate process. I used this to parse the file and then export the structure back to the parent:

#! perl -slw
use strict;
use Storable qw[ freeze ];
use XML::Simple; $XML::Simple::PREFERRED_PARSER = 'XML::Parser'; 

binmode STDOUT;
print freeze XMLin( $ARGV[ 0 ] );

[download]

And then in the main script just a simple backticks command and thaw:

for( 1 .. 1000 ) {
    my $xml = thaw `xmlSto.pl junk.xml`;
    print mem $_; <STDIN>;
}

[download]

It took about 1 extra second or so for the parent process to get access to the data structure, but it cures the leak completely.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

RIP Neil Armstrong

Replies are listed 'Best First'.
Threads / Refcounts
by Bauldric (Novice) on Oct 28, 2012 at 09:42 UTC
    Good morning, my holy reverend

    For my following little program I did no use threads 

        use strict;
    use warnings;

    use CProcess; # own class
    use B;

    CProcess->PrintProcessInfo("perl.exe");

    [download]

    # Output: Process-ID 12280: Memory 13716 KB (+13716 KB) - Peak 13716 KB - Threads 3 - Handles 115 (perl)

    The only packages I use are: 

    use strict;
use warnings;
use Carp;
use FileHandle;
use Sys::Hostname;
use Cwd;
use Win32::Registry;
use IO::Select;
require Win32::Console;
require Win32::Clipboard;
require Win32::PerfLib;
require Win32::OLE;
require Tk::MainWindow;
require 5.000;
require Exporter;

    [download]
    perhaps one of them uses threads.

    To count refcounts I use 

    require B;

    my $anon = [];
    my $otherref = $anon;

    my $refcount =  B::svref_2object($anon)->REFCNT(); # => 2

    [download]

    which counts the referents of the referenced object (and not only of the variable itself like Devel::Peek)

[reply]
[d/l]
[select]
      For my little program

      Is this the 400,000 line/100 package "small program"? :)

      I did not use threads

      I got the notion that you were using threads from your post:

      ID 7772: Memory 34200 KB ( 34200 KB) - Peak 34940 KB - Threads 3 - Han
+dles 124 (perl)
.......................................................^^^^^^^^^

      [download]
      The only packages I use are: ... perhaps one of them uses threads.

      Hm. None of those use threads, (as in Perl Ithreads), that I am aware of.

      But Win32::OLE might well use -- or cause to be used -- kernel threads.

      But more interesting, is that none of those is XML::Simple as is the subject of How can I free the memory of a XML::Simple-Object?

      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.

      RIP Neil Armstrong
[reply]
[d/l]
      Considered using VMMap on your perl process?
[reply]

In Section Seekers of Perl Wisdom