PerlNovice999 has asked for the wisdom of the Perl Monks concerning the following question:

Dear Perl Monks,

There is a memory leak in the following code. I am reading in 28 000 website adresses from one file, then run them through a while loop, extracting some information from them. From what I have read at other nodes, the problem probably lies with an internal call of HTML::FormatText to HTML::Treebuilder, which creates circular references which do not let the variables containing the formatted text of the website scope out when leaving the while-loop.

use warnings; use strict; use diagnostics; use HTML::FormatText; use HTML::TreeBuilder 5 - weak; open INPUT, "< D:/websiteadresses.txt" or die "Problem: $!"; my @INPUT=<INPUT>; close INPUT; while (@INPUT) { my $input=shift(@INPUT); chomp $input; my $content=HTML::FormatText->format_file($input, leftmargin => 0, ri +ghtmargin => 50); # followed by some regular expressions, the results of which are save +d in a new file - all of this is commented out at the moment }

According to the earlier posts this

use HTML::TreeBuilder 5 - weak;

should take care of it - but it does not. The program still runs out of memory after 3000 runs through the while loop

I am now trying to use Object::Destroyer to clear up all objects created by the call to HTML::FormatText. However, I do not understand the relationship between HTML::FormatText and HTML::Treebuilder. The following code will not execute, giving the error message "You should pass an object or constructor". What am I doing wrong here?

use warnings; use strict; use diagnostics; use HTML::FormatText; use HTML::TreeBuilder 5 - weak; open INPUT, "< D:/websiteadresses.txt" or die "Problem: $!"; my @INPUT=<INPUT>; close INPUT; while (@INPUT) { my $input=shift(@INPUT); chomp $input; my $content=HTML::FormatText->format_file($input, leftmargin => 0, ri +ghtmargin => 50); my $sentry = Object::Destroyer->new($content, 'delete' ); # followed by some regular expressions, the results of which are save +d in a new file - all of this is commented out at the moment }

I have also tried this code, based on an earlier node. It executes properly, but still has a memory leak.

use warnings; use strict; use diagnostics; use HTML::FormatText; use HTML::TreeBuilder; use Object::Destroyer; open INPUT, "< D:/websiteadresses.txt" or die "Problem: $!"; my @INPUT=<INPUT>; close INPUT; while (@INPUT) { my $inputfile=shift(@INPUT); chomp $inputfile; print $inputfile; my $formatter = HTML::FormatText->new(); my $content=$formatter->format(Object::Destroyer->new(HTML::TreeBuild +er->new_from_file($inputfile), 'delete')); }

Similarly, this (based on http://www.perl.com/pub/2007/06/07/better-code-through-destruction.html) also still has a memory leak;

use strict; use HTML::TreeBuilder; use Object::Destroyer 2.0; open INPUT, "< D:/websiteadresses.txt" or die "Problem: $!"; my @INPUT=<INPUT>; foreach my $filename (@INPUT) { chomp $filename; print $filename; my $tree = HTML::TreeBuilder->new; my $sentry = Object::Destroyer->new($tree, 'delete'); $tree->parse_file($filename); }

How can I properly destroy the objects created by the HTML::FormatText call to prevent the memory leak?

Any help would be greatly appreciated!

Replies are listed 'Best First'.
Re: Using Object::Destroyer with HTML::FormatText
by Anonymous Monk on Sep 18, 2013 at 08:14 UTC

    Any help would be greatly appreciated!

    And ignored (Re^3: Memory Leak HTML::FormatText)

    Here is another idea, keep a database of stuff you've seen (say DB_File or DBD::SQLite ), and every 1000 urls, restart your process, thus avoiding any memory buildup, without fixing any modules

    When you've processed all urls, quit

      Thank you for your reply!

      Yes, I can simply partition the process into batches of less than 3000 files (i.e., before I run out of memory). Instead of using DBD::SQLite or something like this, I can simply do this by creating seperate files with the website addresses. I am not a computer programmer, I am trying to limit the modules I am using since the CPAN documentation is usually not sufficient for me to understand them...

      I did not ignore the node you are referring to. It just did not help me. I tried the delete call (see my response to the earlier node), but that did not work. I do not know what exactly I have to delete and how to do it. I was hoping that Object::Destroyer would take care of this. I just do not understand how it works exactly when combining it with HTML::FormatText

      I am presumably not the only one using HTML::FormatText with a lot of files and/or using Object::Destroyer, so I am hoping that there is a immediate fix to my problem, rather than using some workaround.

        I can simply do this by creating seperate files with the website addresses.

        don't do this, your computer won't like it very much, stick with sqlite or AnyDBM_File, which ever is easier for you to understand, FWIW, there is always one module like DB_File that you can use that comes with perl, see AnyDBM_File

        I did not ignore the node you are referring to. It just did not help me.

        Why do you think that is? Can you tell me what I suggested?

        I think I suggested adding some Dumper() statements to see what you're dealing with, but the replies to my node included no Dumper() statements or Dumper() output

        Did you see Re^5: Memory Leak HTML::FormatText? Dumper $content? Read that link I gave you?

        I am presumably not the only one using HTML::FormatText with a lot of files and/or using Object::Destroyer, so I am hoping that there is a immediate fix to my problem, rather than using some workaround.

        Sorry PerlNovice999, if I were going post a complete diagnosis with fix I would have done it already. I tried to provide any help, a way (that I have used, and that you can use) to diagnose (and later fix) the problem :)

        Socratic method