Parsing HTML

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Esteemed Monks
I like to strip the html from the webpages from Internet, such that I want to get text and possibly some special fields such as title, date. So I don't get menus, banners, search form etc.. I have to do it for many different sites. Some customization will be required for each site.

On CPAN, I see lots of HTML modules. in HTML section. Which perl modules will help. Please share your experience and wisdom.

Thanks.

Comment on Parsing HTML

Replies are listed 'Best First'.
Re: Parsing HTML by jeffa (Bishop) on Sep 19, 2003 at 15:09 UTC
I would start with HTML::TokeParser::Simple, which is a subclass of HTML::TokeParser (and that module itself is a subclass of HTML::Parser). From the H::TP::S docs: `use HTML::TokeParser::Simple; my $p = HTML::TokeParser::Simple->new( $somefile ); while ( my $token = $p->get_token ) { # This prints all text in an HTML doc (i.e., it strips the HTML) next unless $token->is_text; print $token->as_is; }` [download] From there you allow the tags you do want. See Re: Using a config file in my regexp script. for an example i recently wrote that uses H::TP::S. You can also do a Super Search for `HTML::TokeParser::Simple` and find more examples. Feel free to ask specific questions about the module you do chose to do your dirty work. :) jeffa L-LL-L--L-LL-L--L-LL-L-- -R--R-RR-R--R-RR-R--R-RR B--B--B--B--B--B--B--B-- H---H---H---H---H---H--- (the triplet paradiddle with high-hat)	[reply] [d/l]
Re: Parsing HTML by jdtoronto (Prior) on Sep 19, 2003 at 15:41 UTC
Requests for assistance on parsing HTML are very common here in the monastery - so try doing a Super-Search - I am sure you will come up with a lot. Besides jeffa's recomendation I would also suggest you look at HTML::TreeBuilder and the somewhat useful text: Perl & LWP by Sean Burke, from O'Reilly. Good luck! jdtoronto	[reply]
Re: Parsing HTML by BUU (Prior) on Sep 19, 2003 at 18:22 UTC
Non-perl, but what about `lynx -dump` ? Granted you have to fiddle with the output a tad but that gives you nice rendered text without any html..	[reply] [d/l]