Re: Cleaning up HTML

It is certainly not smaller, and is probably slower (haven't done any benchmarks), but it is very flexible and powerful: try using HTML::StripScripts via HTML::StripScripts::Parser.

This will churn through your HTML (either a complete HTML page or an HTML snippet), tidy up the HTML, fix tag nesting, remove scripts, remove unknown attributes etc.

Through the Rules => {} parameter, you can specify exactly what tags and attributes you want to allow through, adding regexes or callbacks to customise the results.

fixing tag soup, like "foo"
Yes
avoid inline elements wrapped around block elements, for example a "p" tag wrapped in "font" tags
Yes
stripping (some) empty or whitespace-only elements, such as ""
Yes with a callback such as:
 Rules => {
 * => sub {
 my ($filter,$element) = @_;
 return $element->{content} =~ m/\S/ ? 1 : 0
 }
 }
[download]
removing unnecessary tags, for example, if there's a "" tag, strip it and its corresponding "" tag, because that's my default font and size for the table - but leave in a font tag if there are any other attributes after dropping the ones with the default value: for example "" -> ""
with a callback like:
 Rules => {
 font => sub {
 my ($filter,$element) = @_;
 my $attr = $element->{attr};
 delete $attr->{size} if $attr->{size} && $attr->{
+size} eq 1;
 delete $attr->{face} if $attr->{face} && lc($attr
+->{face}) eq 'verdana';
 return keys %$attr ? 1 : 0;
 }
 }
[download]
moving " " tags out of links, when at the edge of the link text: <a href="linkto">link text </a> -> <a href="linkto">link text</a> 
This is the only one I don't have a callback for. It would need to be handled with a regex run at the end. That said, it'd be pretty easy to change sub output_stack_entry to allow you to pass back the literal HTML to use instead of reassembling the components, for instance:
 Rules => {
 a => sub {
 my ($filter,$element) = @_;
 my $content = $element->{content};
 my ($pre,$post) = ('','');
 if ($content=~s{^\s* }{}) {
 $pre = ' ';
 }
 if ($content=~s{ \s*$}{}) {
 $post = ' ';
 }
 if ($pre || $post) {
 $element->{literal} = 
 $pre.
 '<a ' . $self->_hss_join_attribs( $eleme
+nt{attr} ) . '>'
 . $content 
 . '</a>'
 . $post;
 }
 return 1;
 }
 }
[download]

Clint

Update: I'm the maintainer of HTML::StripScripts, and I added the Rules => {} parameter to HTML::StripScripts, which makes it easier to customise. But all credit for the underlying module must go the Nick Cleaton, the original author, who did a very very good job indeed

Update: Tidied up the HTML

Comment on Re: Cleaning up HTML Select or Download Code

Replies are listed 'Best First'.
Re^2: Cleaning up HTML by bart (Canon) on Apr 22, 2008 at 09:30 UTC
I've been trying some things with this module, and it's a bit hard going. Now, how do you recommend combining two `span` tags? For example, I have this crufty html: `<span style="font-family:Arial"><span style="font-size:10pt; color:#00 +0080;">text</span></span>` [download] I'd like to combine the two span tags, merging their style attributes: `<span style="font-family:Arial; font-size:10pt; color:#000080;">text</ +span></span>` [download] Well, I know HTML::StripScripts has built in handling of style tags... Now, if I do a callback for the `span` like `my $p = HTML::StripScripts::Parser->new({ Rules => { span => sub { my ($filter,$element) = @_; print Dumper $element if $element->{content} =~ /^ +<span\W/; 1; }, } } );` [download] then I get this result for the outer span: `$VAR1 = { 'content' => '<span style="font-size:10pt; color:#000080;">t +ext</span>', 'tag' => 'span', 'attr' => { 'style' => 'font-family:Arial' } };` [download] How do you recommend to proceed from here? Should I parse the "content" again, and how? Also... how do you remove tags but not its content? If I return '0' from this callback sub, then both the tags and the inner HTML are gone.	[reply] [d/l] [select]
Re^3: Cleaning up HTML by clinton (Priest) on Apr 22, 2008 at 09:51 UTC
I've been trying some things with this module, and it's a bit hard going. Agreed - it makes certain things very easy, but other bits (like adding allowed attributes) are nasty and crust. As we've discussed privately, this module could do with a major API overhaul. And a new name! Stripping scripts is just a part of what this does. It's actually a very powerful HTML tidier I'd like to combine the two span tags, merging their style attributes: Merging adjacent tags wasn't a use that I envisaged, but that was my own lack of imagination - the next version will have some nice way of querying the parent and sibling elements. I've been trying to think about how you could do this with the current module (using private methods and properties), but it's nasty, and would probably result in having to reparse content, because to merge two `span`s, you need to know: that the child `span` HAS content the the parent `span` has NO content but you don't know that until you're finished with the parent `span` because it's a stream parser by which time, the child `span` has been converted to text As per the private message you've just sent me, maintaining the nodes as a DOM-tree seems like the only reasonable way to handle this. UPDATE So if we're planning on building DOM-trees, this starts sounding like a job for XML::LibXML, or HTML::TreeBuilder. But HTML::StripScripts does three things that neither of these modules does (AFAIK): it understands the meaning and context of tags, so that it can figure out that lone `<li>` elements cannot exist outside `<ul>` or `<ol>` it provides a mechanism for parsing / allowing / disallowing attribute and CSS style declarations it cleans up XSS attacks Ideas for the new glamorous DOM-tree based version gladly accepted	[reply] [d/l] [select]