Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number

Re: HTML::Strip Problem

by tachyon (Chancellor)
on Mar 29, 2004 at 04:53 UTC ( #340509=note: print w/replies, xml ) Need Help??

in reply to HTML::Strip Problem

We use something like this:

use HTML::Parser 3; use LWP::Simple; my $html = get(""); print body_text($html); sub body_text { my $content = $_[0] || return 'EMPTY BODY'; # HTML::Parser is broken on Javascript and styles # (well it leaves it in the text) so we 'fix' it.... my $p = HTML::Parser->new( start_h => [ sub{ $_[0]->{text}.=' '; $_[0]->{skip}++ if $_[1] + eq 'script' or $_[1] eq 'style'; } , 'self,tag' ], end_h => [ sub{ $_[0]->{skip}-- if $_[1] eq '/script' or $_[ +1] eq '/style'; } , 'self,tag' ], text_h => [ sub{ $_[0]->{text}.=$_[1] unless $_[0]->{skip}}, +'self,dtext' ] )->parse($content); $p->eof(); my $text = $p->{text}; # remove escapes $text =~ s/&nbsp;/ /gi; $text =~ s/&[^;]+;/ /g; # remove non ASCII printable chars, leaves punctuation stuff $text =~ s/[^\040-\177]+/ /g; # remove any < or > in case parser choked - rare but happens $text =~ s/[<>]/ /g; # crunch whitespace $text =~ s/\s{2,}/ /g; $text =~ s/^\s+//g; return $text; }

Hint. Using LWP::Simple to get web pages for a search engine is doomed to failure. You will for a start collect the frameset but not the frames of every page you visit that uses frames. You will be blocked by a number of sites for not being IE. You will ignore metarefreshes and 302 found (perhaps you want to perhaps not). You also have no idea of the problem if you don't get content.



Replies are listed 'Best First'.
Re: Re: HTML::Strip Problem
by mkurtis (Scribe) on Mar 29, 2004 at 05:23 UTC
    thanks for the code tachyon. Thats not my crawler above there. I used LWP::Simple only to get html to see how well strip would parse it. Thanks for the tip though. My real crawler or something close, i might not have updated is in the code section.

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://340509]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (3)
As of 2023-02-04 02:57 GMT
Find Nodes?
    Voting Booth?
    I prefer not to run the latest version of Perl because:

    Results (30 votes). Check out past polls.