PerlNovice999 has asked for the wisdom of the Perl Monks concerning the following question:
Dear Perl Monks,
There is a memory leak in the following code. I am reading in 28 000 website adresses from one file, then run them through a while loop, extracting some information from them. From what I have read at other nodes, the problem probably lies with an internal call of HTML::FormatText to HTML::Treebuilder, which creates circular references which do not let the variables containing the formatted text of the website scope out when leaving the while-loop.
use warnings; use strict; use diagnostics; use HTML::FormatText; use HTML::TreeBuilder 5 - weak; open INPUT, "< D:/websiteadresses.txt" or die "Problem: $!"; my @INPUT=<INPUT>; close INPUT; while (@INPUT) { my $input=shift(@INPUT); chomp $input; my $content=HTML::FormatText->format_file($input, leftmargin => 0, ri +ghtmargin => 50); # followed by some regular expressions, the results of which are save +d in a new file - all of this is commented out at the moment }
According to the earlier posts this
use HTML::TreeBuilder 5 - weak;
should take care of it - but it does not. The program still runs out of memory after 3000 runs through the while loop
I am now trying to use Object::Destroyer to clear up all objects created by the call to HTML::FormatText. However, I do not understand the relationship between HTML::FormatText and HTML::Treebuilder. The following code will not execute, giving the error message "You should pass an object or constructor". What am I doing wrong here?
use warnings; use strict; use diagnostics; use HTML::FormatText; use HTML::TreeBuilder 5 - weak; open INPUT, "< D:/websiteadresses.txt" or die "Problem: $!"; my @INPUT=<INPUT>; close INPUT; while (@INPUT) { my $input=shift(@INPUT); chomp $input; my $content=HTML::FormatText->format_file($input, leftmargin => 0, ri +ghtmargin => 50); my $sentry = Object::Destroyer->new($content, 'delete' ); # followed by some regular expressions, the results of which are save +d in a new file - all of this is commented out at the moment }
I have also tried this code, based on an earlier node. It executes properly, but still has a memory leak.
use warnings; use strict; use diagnostics; use HTML::FormatText; use HTML::TreeBuilder; use Object::Destroyer; open INPUT, "< D:/websiteadresses.txt" or die "Problem: $!"; my @INPUT=<INPUT>; close INPUT; while (@INPUT) { my $inputfile=shift(@INPUT); chomp $inputfile; print $inputfile; my $formatter = HTML::FormatText->new(); my $content=$formatter->format(Object::Destroyer->new(HTML::TreeBuild +er->new_from_file($inputfile), 'delete')); }
Similarly, this (based on http://www.perl.com/pub/2007/06/07/better-code-through-destruction.html) also still has a memory leak;
use strict; use HTML::TreeBuilder; use Object::Destroyer 2.0; open INPUT, "< D:/websiteadresses.txt" or die "Problem: $!"; my @INPUT=<INPUT>; foreach my $filename (@INPUT) { chomp $filename; print $filename; my $tree = HTML::TreeBuilder->new; my $sentry = Object::Destroyer->new($tree, 'delete'); $tree->parse_file($filename); }
How can I properly destroy the objects created by the HTML::FormatText call to prevent the memory leak?
Any help would be greatly appreciated!
|
|---|
| Replies are listed 'Best First'. | |
|---|---|
|
Re: Using Object::Destroyer with HTML::FormatText
by Anonymous Monk on Sep 18, 2013 at 08:14 UTC | |
by PerlNovice999 (Novice) on Sep 18, 2013 at 09:05 UTC | |
by Anonymous Monk on Sep 18, 2013 at 10:40 UTC |