in reply to Re^2: XML::Twig loves to eat my memory
in thread XML::Twig loves to eat my memory

twig_roots does prove to be most useful for when you only want one little bit of data. A single file only consumes 19MB as opposed to 166MB. However it still does not release memory when I try to $t->dispose(). This also does not solve my problem for scripts that have to process all the data.

Replies are listed 'Best First'.
Re^4: XML::Twig loves to eat my memory
by ikegami (Patriarch) on Jul 22, 2010 at 22:42 UTC

    However it still does not release memory when I try to $t->dispose().

    How did you ascertain that?

[reply]
      By watching the size of the process. It does not decrease after dispose is called. Unless I misunderstand the purpose of dispose I would expect memory to be released and the size of the process to go down.
[reply]

        Unless I misunderstand the purpose of dispose I would expect memory to be released...

        Yes. (Actually, it sounds like it's doesn't do anything on a system will a properly installed Scalar::Util because it's done automatically then.)

        ...and the size of the process to go down.

        That does not follow. Freeing memory does not necessarily make the process size go down. It's only guaranteed to go back to Perl's free memory pool. In some circumstances on some OSes, the process size can go down, but there's no guarantee of it.

        For example, you could observe the following: 

        my $x;
$x .= "a" for 1..1024;  # Process size goes up
undef $x;               # Process size doesn't go down
$x .= "a" for 1..1024;  # Process size doesn't go up

        [download]

        undef $x; freed the string's buffer — observable with Devel::Peek — but it might not release it to the OS. However, Perl will now be able to reuse that memory to rebuild $x.

        To see if the memory is freed, what you need to do is use some memory and see if the process goes up. Do as almut suggested and call process twice in a row, checking the process size after each call.

[reply]
[d/l]
[select]
Re^4: XML::Twig loves to eat my memory
by AndyZaft (Hermit) on Jul 22, 2010 at 21:54 UTC
    Very strange, since dispose calls the Twig object's destructor explicitly. But then again I don't know much about how the GC works in the Perl process. So your memory usage keeps growing the more files you process if I'm understanding this correctly, right?
[reply]
      Correct, no matter what I have tried it keeps growing approx 160-180MB for each file.
[reply]

In Section Seekers of Perl Wisdom