http://qs1969.pair.com?node_id=554219


in reply to Simplify HTML programatically

I use SAX for this. For me it provides the best performance/maintainability ratio.

Here is a standalone program that takes a html file name as an argument and prints the output to STDOUT.

Its purpose is to strip all tags except p, div, and a tags.

The program driver sets up the SAX pipeline. It uses XML::Driver::HTML as the SAX driver and XML::SAX::Writer as the writer.

When parse is called on the driver, the Pipeline sends the driver's stream to some helper modules, to our filter module, and finally the writer.

In the custom filter, start_element is called for each tag. The code checks to see if the tag is either an a, div, or p tag, and if it is, it forwards the tag to the writer. Otherwise, the tag is ignored and removed from the stream.

The same work needs to happen in the end_element callback.

use warnings; use strict; use XML::SAX::Machines qw(Pipeline); use XML::Driver::HTML; use XML::Filter::SAX1toSAX2; use XML::Filter::BufferText; use XML::SAX::Writer; my $output; # transformation target my $writer = XML::SAX::Writer->new( Output => \$output ); my $machine = Pipeline( 'XML::Filter::SAX1toSAX2' => 'XML::Filter::BufferText' => 'XML::Filter::HtmlTagStripper' => $writer ); my $html = new XML::Driver::HTML( Handler => $machine, Source => { SystemId => $ARGV[0] } ); $html->parse(); print $output; package XML::Filter::HtmlTagStripper; use base qw|XML::SAX::Base|; # <marker language="foo" /> # $el->{Name} == 'marker' # $el->{Attributes}{'{}language'} == language attribute # $el->{Attributes}{'{}language'}{Value} == 'foo' sub start_element { my($self, $el) = @_; if ( $el->{Name} =~ m/^(?:p|div|a)$/i ) { $self->SUPER::start_element( $el ); } } sub end_element { my($self, $el) = @_; if ( $el->{Name} =~ m/^(?:p|div|a)$/i ) { $self->SUPER::end_element( $el ); } } 1;

I know its not the most un-rocket-sciencey thing in the world, but its not too tricky, and once it clicks in your head and you realize that this is high performance xml parsing, the possibilities are boggling.

Lets take a look at a run:

$ cat striptags.html <html> <head> <title>Test Document</title> </head> <body> <p>The first paragraph</p> <p>the second paragraph</p> <hr width="75%"> <div>last modified: WHENEVER</div> </body> </html> $ perl striptags.pl striptags.html Test Document<p>The first paragraph</p><p>the second paragraph</p><div +>last modified: WHENEVER</div>
Enjoy, Todd W.