I have written a code which extracts all the words from an html page, according to some heuristics. the output is correct in this my code, but there is some mysterious memory leak. I am really trying to hone some of my perl skills, but this has me banging my head.
the input is a long long file with many html pages, separated by the http header and some additional information. Some of the code may be a little weird, its kind of slapped together, but the output is what i want. without further ado:
use strict;
use warnings;
use HTML::Parse;
use HTML::FormatText;
use File::Slurp;
use Lingua::Stem::Snowball;
$/ = 'warc/0.9 ';
my $sep = 'warc/0.9 ';
open FILE, shift;
open OUT, '>', shift;
open DATA, '>', shift;
my $total = 0;
my $stemmer = Lingua::Stem::Snowball->new( lang => 'en' );
while(<FILE>)
{
my $line = $_;
$line =~ s/$sep//;
if($line)
{
$line =~ /(\d+)\sresponse\s(\S+)\s/i;
my $id = $1;
my $url = $2;
#print "$1, $2\n+++++\n";
$line =~ s/[^<]*//; #remove everything up to the 1st html tag
+ (header, etc)
my $len = length($line);
if($len < 600000)
{
print DATA "$id\t$url\t$total\t$len\n";
$total += $len;
#print OUT "$line";
my $plain_text = HTML::FormatText->new->format(parse_html(
+$line));
$plain_text =~ s/\[image\]//ig;
$plain_text =~ s/(\S)/\L$1/ig;
my @words = $plain_text =~ /\b\S+\b/ig;
#my $stemmer = Lingua::Stem::Snowball->new( lang => 'en' )
+;
$stemmer->stem_in_place( \@words );
foreach my $x (@words)
{
if($x =~ /^[A-Za-z0-9]+$/ and $x !~ /(http:\/\/(\S+)\
+b)|(&?nbsp)|(\b.*\d{5,
}.*\b)|(
^\d+$)|(\S{32,})/)
{
print OUT "$x ";
}
}
print OUT "\n";
print OUT '+-+-+-+-+', "\n";
}
}
}
can any monks give me suggestions here?
Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
Read Where should I post X? if you're not absolutely sure you're posting in the right place.
Please read these before you post! —
Posts may use any of the Perl Monks Approved HTML tags:
- a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
| |
For: |
|
Use: |
| & | | & |
| < | | < |
| > | | > |
| [ | | [ |
| ] | | ] |
Link using PerlMonks shortcuts! What shortcuts can I use for linking?
See Writeup Formatting Tips and other pages linked from there for more info.