I have written a code which extracts all the words from an html page, according to some heuristics. the output is correct in this my code, but there is some mysterious memory leak. I am really trying to hone some of my perl skills, but this has me banging my head.

the input is a long long file with many html pages, separated by the http header and some additional information. Some of the code may be a little weird, its kind of slapped together, but the output is what i want. without further ado:
use strict; use warnings; use HTML::Parse; use HTML::FormatText; use File::Slurp; use Lingua::Stem::Snowball; $/ = 'warc/0.9 '; my $sep = 'warc/0.9 '; open FILE, shift; open OUT, '>', shift; open DATA, '>', shift; my $total = 0; my $stemmer = Lingua::Stem::Snowball->new( lang => 'en' ); while(<FILE>) { my $line = $_; $line =~ s/$sep//; if($line) { $line =~ /(\d+)\sresponse\s(\S+)\s/i; my $id = $1; my $url = $2; #print "$1, $2\n+++++\n"; $line =~ s/[^<]*//; #remove everything up to the 1st html tag + (header, etc) my $len = length($line); if($len < 600000) { print DATA "$id\t$url\t$total\t$len\n"; $total += $len; #print OUT "$line"; my $plain_text = HTML::FormatText->new->format(parse_html( +$line)); $plain_text =~ s/\[image\]//ig; $plain_text =~ s/(\S)/\L$1/ig; my @words = $plain_text =~ /\b\S+\b/ig; #my $stemmer = Lingua::Stem::Snowball->new( lang => 'en' ) +; $stemmer->stem_in_place( \@words ); foreach my $x (@words) { if($x =~ /^[A-Za-z0-9]+$/ and $x !~ /(http:\/\/(\S+)\ +b)|(&?nbsp)|(\b.*\d{5, }.*\b)|( ^\d+$)|(\S{32,})/) { print OUT "$x "; } } print OUT "\n"; print OUT '+-+-+-+-+', "\n"; } } }
can any monks give me suggestions here?

In reply to Memory Leak? i'm clueless. by downer

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.