HTML::TokeParser not stripping entities and xhtml

bradcathey has asked for the wisdom of the Perl Monks concerning the following question:

Fellow monasterians,

New to HTML::Tokeparser (or any parser, for that matter).

I'm pulling xhtml-formatted content from a database for display in a browser. The browser is not rendering entities, in particular, those that follow tags that are not being rendered, namely closing xhtml tags that have no opening tags (badly formatted, I know, but I have no control over that).

So, I running the content through HTML::Tokeparser first but it's not stripping out the delinquent tags.

Does the parser need the opening tag to work? Short of regex's, is there another way to do this? What am I missing? TIA!

use HTML::TokeParser;

my $result;
my $p = HTML::TokeParser->new(\$text);

while ( my $token = $p->get_token ) {
    if ($token->[0] eq 'T') { 
    $result.= $token->[1];
}
[download]

Before parsing:

</tr />
</tbody />
</table /></p>
<p>We have&nbsp;different groups to help you through the buying proces
+s.&nbsp;Our team of counselors and volunteers can provide transportat
+ion, and childcare.&nbsp;</p>
<p>&nbsp;</p>';
[download]

After parsing:

</tr />
</tbody />
</table />
We have&nbsp;different groups to help you through the buying process.&
+nbsp;Our team of counselors and volunteers can provide transportation
+, and childcare.&nbsp;
&nbsp;';
[download]

Note: It's getting the <p> tags.

—Brad
"The important work of moving the world forward does not wait to be done by perfect men." George Eliot

Comment on HTML::TokeParser not stripping entities and xhtml Select or Download Code

Replies are listed 'Best First'.
Re: HTML::TokeParser not stripping entities and xhtml by tphyahoo (Vicar) on Dec 02, 2005 at 16:57 UTC
Maybe you should be using XML::TokeParser or XML::Twig or at any rate, some XML parser. Since your input is XML. Html is always xml, but not the other way around. And xhtml is xml, not html.	[reply]
Re: HTML::TokeParser not stripping entities and xhtml by merlyn (Sage) on Dec 02, 2005 at 16:59 UTC
If that's your entire text, it's not well-formed. I wouldn't expect a parser to work correctly on a snippet like that. I know that HTML::Parser (on which HTML::TokeParser is based) insists on delivering well-matched pairs of start and end tags. I talk about that in a couple of my columns. -- Randal L. Schwartz, Perl hacker Be sure to read my standard disclaimer if this is a reply.	[reply]
Re: HTML::TokeParser not stripping entities and xhtml by Ovid (Cardinal) on Dec 02, 2005 at 17:36 UTC
Even though HTML::TokeParser is great, you can alleviate some pain by switching to HTML::TokeParser::Simple. If you just want to strip out the malformed tags, here's a first try: #!/usr/bin/perl use strict; use warnings; use HTML::TokeParser::Simple; my $text = '</tr /> </tbody /> </table /></p> <p>We have different groups to help you through the buying proces +s. Our team of counselors and volunteers can provide transportat +ion, and childcare. </p> <p> </p>'; my $result = ''; my $p = HTML::TokeParser::Simple->new(\$text); while ( my $token = $p->get_token ) { my $text = $token->as_is; if ($token->is_tag) { next if $text =~ /^<\/.*\/>$/; } $result .= $text; } print $result; [download] Cheers, Ovid New address of my CGI Course.	[reply] [d/l]