We use something like this:

use HTML::Parser 3;
use LWP::Simple;
my $html = get("http://perlmonks.org");
print body_text($html);


sub body_text {
    my $content = $_[0] || return 'EMPTY BODY';

    # HTML::Parser is broken on Javascript and styles 
    # (well it leaves it in the text) so we 'fix' it....

    my $p = HTML::Parser->new( 
        start_h => [ sub{ $_[0]->{text}.=' '; $_[0]->{skip}++ if $_[1]
+ eq 'script' or $_[1] eq 'style'; } , 'self,tag' ],
        end_h   => [ sub{ $_[0]->{skip}-- if $_[1] eq '/script' or $_[
+1] eq '/style'; } , 'self,tag' ], 
        text_h  => [ sub{ $_[0]->{text}.=$_[1] unless $_[0]->{skip}}, 
+'self,dtext' ]
        )->parse($content);
    
    $p->eof();
    my $text = $p->{text};
    
    # remove escapes
    $text =~ s/ / /gi;
    $text =~ s/&[^;]+;/ /g;

    # remove non ASCII printable chars, leaves punctuation stuff
    $text =~ s/[^\040-\177]+/ /g;

    # remove any < or > in case parser choked - rare but happens
    $text =~ s/[<>]/ /g;

    # crunch whitespace
    $text =~ s/\s{2,}/ /g;
    $text =~ s/^\s+//g;

  return $text;
}
[download]

Hint. Using LWP::Simple to get web pages for a search engine is doomed to failure. You will for a start collect the frameset but not the frames of every page you visit that uses frames. You will be blocked by a number of sites for not being IE. You will ignore metarefreshes and 302 found (perhaps you want to perhaps not). You also have no idea of the problem if you don't get content.

thanks for the code tachyon. Thats not my crawler above there. I used LWP::Simple only to get html to see how well strip would parse it. Thanks for the tip though. My real crawler or something close, i might not have updated is in the code section.

As for why the second snippet only produces about 3/4 of the expected text output, that might be a matter of a "syntax error" in the yahoo HTML source. (But how could Yahoo make a mistake like that?? I'm shocked! Shocked!!) Anyway, it appears that HTML::Strip does not do syntax checking (so it probably won't generate parsing errors that you can trap), and there may be some stray angle brackets or flubbed entities in the source text (perhaps 3/4 of the way into the file) that are causing trouble. You would need to just probe the text to see if that's what the problem is -- e.g. run a validating parser on it, or simply try out some simple one-liners that will isolate angle brackets and/or ampersands, along with the things adjacent to them...

thanks graff. I did try placing the text into $file but it only contained the first line of the file when i did it. When i made it an array however it printed out the whole file. Im not sure why that is, but thats why i used @file. I think ill just use tachyon's parser as that seems to fix my previous problems with parser.

#!/usr/bin/perl

use strict;
use warnings;

use HTML::Strip;

my $hs = HTML::Strip->new();
my @file;
open(FILE,"<test.html") || die "Cannot open file: $!";
while(<FILE>)
{
  print $hs->parse($_);
}
close(FILE);
[download]