Re: Unicode nightmare: is there a cheat sheet or solutions diagram?

If that's any comfort, you are probably not the only one to have been tortured by Unicode for that long. I have a last-resort shell trick that goes iconv -f utf8 -t utf8 <dodgy_file> || iconv -f iso-8859-1 -t utf8 <dodgy_file> > <utf8_file> for those cases where I am not 100% sure of the encoding I get, and I don't have the time to debug it properly.

In your case I wonder if the problem is not that you have some non Latin1 characters in your input (that got there through entities in the HTML maybe, something like — for example.

My machine has got UTF-8 locale, so I can't reproduce exactly your problem, but the 2 one liners below give me outputs in different encodings:

perl -MHTML::TreeBuilder::XPath -e'$t= HTML::TreeBuilder::XPath->new;
$t->parse( "<html><body><p>para &copy;</p></body></html>");
$p=$t->findvalue( "//p"); 
print $p, "\n";'
[download]

That first one gives me a result in latin1.

perl -MHTML::TreeBuilder::XPath -e'$t= HTML::TreeBuilder::XPath->new;
$t->parse( "<html><body><p>para &copy; &mdash;</p></body></html>");
@p=$t->findnodes( "//p"); 
print $p[0]->as_text(), "\n";'
[download]

I just added an —, an entity that cannot be printed in Latin1, to the input et voilà! Now the result is in UTF-8.

In that case using HTML::Entities should help:

perl -MHTML::TreeBuilder::XPath -MHTML::Entities -e'
 $t= HTML::TreeBuilder::XPath->new;
 $t->parse( "<html><body><p>para &copy; &mdash;</p></body></html>"); 
 @p=$t->findnodes( "//p");
 $out= $p[0]->as_text; 
  encode_entities( $out);
 print $out, "\n";'
[download]

Comment on Re: Unicode nightmare: is there a cheat sheet or solutions diagram? Select or Download Code