Re: Encoding/decoding question

Replies are listed 'Best First'.
Re^2: Encoding/decoding question by slugger415 (Monk) on Sep 11, 2011 at 19:13 UTC
aha! interesting -- I actually used tidy.exe to convert the HTML to XHTML, so tidy must be the culprit. Thanks for the tip! (know any better way to turn HTML into XHTML? maybe I should just be using a Perl HTML parser...)	[reply]
Re^3: Encoding/decoding question by ikegami (Patriarch) on Sep 11, 2011 at 20:26 UTC
I doubt that. I suspect the HTML was buggy too. Could you show the HTML's HEAD element and the `od -c` output for `réserve`? ( Update: hum, .exe? You might not have `od`. Alternative: `perl -nE"say unpack 'H*', $_ if /serv/;" file.html` ) By the way, XML::LibXML has functions for parsing HTML.	[reply] [d/l] [select]
Re^4: Encoding/decoding question by tchrist (Pilgrim) on Sep 12, 2011 at 00:43 UTC
I doubt that. I suspect the HTML was buggy too. Could you show the HTML's HEAD element and the `od -c` output for `réserve`? ( Update: hum, .exe? You might not have `od`. Alternative: `perl -nE"say unpack 'H', $_ if /serv/;" file.html` ) I once again recommend the uniquote* program for such things. It is really way better than `od` or `cat -v` or anything, because it actually shows you the proper characters. $ perl -Mutf8 -CS -wle 'print "réserve"' \| uniquote r\N{U+E9}serve $ perl -Mutf8 -CS -wle 'print "réserve"' \| uniquote -x r\x{E9}serve $ perl -Mutf8 -CS -wle 'print "réserve"' \| uniquote -v r\N{LATIN SMALL LETTER E WITH ACUTE}serve $ perl -Mutf8 -CS -wle 'print "réserve"' \| uniquote -b r\xC3\xA9serve $ perl -Mutf8 -CS -wle 'print "réserve"' \| uniquote --xml réserve $ perl -Mutf8 -CS -wle 'print "réserve"' \| uniquote --html réserve $ perl -Mutf8 -CS -wle 'print "réserve"' \| uniquote --html --verbose réserve $ perl -Mutf8 -CS -wle 'print "réserve"' \| nfd \| uniquote -v re\N{COMBINING ACUTE ACCENT}serve $ perl -Mutf8 -CS -wle 'print "réserve"' \| iconv -f UTF-8 -t UTF-16 \| +uniquote --encoding=UTF-16 -x r\x{E9}serve $ perl -Mutf8 -CS -wle 'print "réserve"' \| iconv -f UTF-8 -t UTF-16 \| +uniquote -b \xFE\xFF\x00r\x00\xE9\x00s\x00e\x00r\x00v\x00e\x00 $ perl -Mutf8 -CS -wle 'print "réserve"' \| iconv -f UTF-8 -t MacRoman +\| uniquote --encoding=MacRoman -x r\x{E9}serve $ perl -Mutf8 -CS -wle 'print "réserve"' \| iconv -f UTF-8 -t MacRoman +\| uniquote -b r\x8Eserve $ perl -Mutf8 -CS -wle 'print "réserve"' > reserve.utf8 $ iconv -f UTF-8 -t MacRoman < reserve.utf8 > reserve.macroman $ iconv -f UTF-8 -t UTF16-BE < reserve.utf8 > reserve.utf16be $ uniwc reserve.{macroman,utf8,utf16be} Paras Lines Words Graphs Chars Bytes File 0 1 1 8 8 8 reserve.macroman 0 1 1 8 8 9 reserve.utf8 0 1 1 8 8 16 reserve.utf16be $ uniquote reserve.{macroman,utf8,utf16be} r\N{U+E9}serve r\N{U+E9}serve r\N{U+E9}serve $ uniquote -b reserve.{macroman,utf8,utf16be} r\x8Eserve r\xC3\xA9serve \x00r\x00\xE9\x00s\x00e\x00r\x00v\x00e\x00 [download] See how nifty that is?	[reply] [d/l]
Re^5: Encoding/decoding question by ikegami (Patriarch) on Sep 12, 2011 at 02:35 UTC
Re^6: Encoding/decoding question by tchrist (Pilgrim) on Sep 12, 2011 at 20:09 UTC
Re^5: Encoding/decoding question by Anonymous Monk on Sep 12, 2011 at 20:34 UTC
Re^4: Encoding/decoding question by slugger415 (Monk) on Sep 12, 2011 at 15:11 UTC
I see tidy.exe is giving me this message: Character codes 128 to 159 (U+0080 to U+009F) are not allowed in HTML; even if they were, they would likely be unprintable control characters. Tidy assumed you wanted to refer to a character with the same byte value in the specified encoding and replaced that reference with the Unicode equivalent. Here's the very top of the original (pre-tidy'd) HTML file (from our friend the facebook) <!DOCTYPE HTML> <html class=" videoCallEnabled" id="facebook" lang="en"><head> <meta http-equiv="content-type" content="text/html; charset=UTF-8"> <meta charset="utf-8"><script>CavalryLogger=false;window._script_path = "\/home.php";window._EagleEyeSeed="Nq0j";</script><noscript> <meta http-equiv="refresh" content="0; URL=/?_fb_noscript=1" /> </noscript> <meta name="robots" content="noodp,noydir"> ... followed by loads of scripts and stylesheets. The output from your command above, run on the html file, produced thousands of characters such as: 3c6c696e6b2068726566... Not sure if you're looking for anything in particular. Thanks for your help, Scott	[reply]
Re^5: Encoding/decoding question by tchrist (Pilgrim) on Sep 12, 2011 at 15:51 UTC
Re^6: Encoding/decoding question by slugger415 (Monk) on Sep 12, 2011 at 20:20 UTC
Some notes below your chosen depth have not been shown here
Re^6: Encoding/decoding question by slugger415 (Monk) on Sep 12, 2011 at 20:30 UTC
Some notes below your chosen depth have not been shown here
Re^3: Encoding/decoding question by mirod (Canon) on Sep 13, 2011 at 08:36 UTC
You can use HTML::TreeBuilder to parse the HTML, then output it in XHTML, using the `as_XML` method, which works most of the time. It may not help with the encoding problem though, especially if the HTML lies about its encoding. XML::Twig can do this for you BTW, so in fact you may not need to use `tidy` at all, just install HTML::TreeBuilder and then use `parsefile_html` to parse the HTML. Also HTML::Tidy uses a fork of tidy, and may be worth a try.	[reply]