UTF-8 and browsers

amonroy has asked for the wisdom of the Perl Monks concerning the following question:

Sorry if this is off-topic, but I need some advice from people who have worked with UTF-8.

Do you know why Firefox (so far I have tested under Windows, Linux and MacOS X) does not show an Ö when I send the following data to the browser:

print $cgi->header(-charset => "utf-8"), "\x{4F}\x{CC}\x{88}";

Firefox shows an O followed by the dieresis instead of a "clean" O with dieresis. Other browsers such as IE (under MacOS and WinXP) and Safari handle this correctly.

Update

The sequence \x{4F}\x{CC}\x{88} represents the Unicode LATIN CAPITAL LETTER O followed by a DIAERESIS, which in theory should be displayed just like unicode LATIN CAPITAL LETTER O WITH DIAERESIS (\x{C3}\x{96}).

But I found that Firefox might have a bug displaying Unicode canonical equivalents.

Now, I have a less off-topic question: How can I convert the Unicode LATIN CAPITAL LETTER O followed by a DIAERESIS to LATIN CAPITAL LETTER O WITH DIAERESIS. Or in hexadecimal terms, how can I convert \x{4F}\x{CC}\x{88} to \x{C3}\x{96}?

I tried using all the normalization forms of Unicode::Normalizer, but no luck.

Comment on UTF-8 and browsers - Update Download Code

Replies are listed 'Best First'.
Re: UTF-8 and browsers - Update by theorbtwo (Prior) on Feb 14, 2005 at 06:35 UTC
In general, you shouldn't try to utf8-encode yourself -- instead, let perl do the work of encoding, and just give the abstract codepoints. This is, in fact, what is causing your problem. You specified character 0x4F, followed by character 0x308, that is, "LATIN CAPITAL LETTER O", "COMBINING DIAERESIS". I think you wanted that to be 0xA8, "DIAERESIS". Warning: Unless otherwise stated, code is untested. Do not use without understanding. Code is posted in the hopes it is useful, but without warranty. All copyrights are relinquished into the public domain unless otherwise stated. I am not an angel. I am capable of error, and err on a fairly regular basis. If I made a mistake, please let me know (such as by replying to this node).	[reply]
Re: UTF-8 and browsers - Update by dakkar (Hermit) on Feb 14, 2005 at 12:39 UTC
Bug in Firefox. It should work as you describe. As for the composition: first of all, work on characters, or at least or codepoints, not on utf-8 bytes. Second, you want Unicode Normal Form C (see Unicode::Normalize), so that you can write: `use Unicode::Normalize; use charnames ':full'; # this is just to make things easier in this ex +ample binmode(STDOUT,':utf8'); # this to make 'print' output utf-8 bytes my $a="O\N{COMBINING DIAERESIS}"; my $b=NFC($a); print length($a),$a,"\n"; print length($b),$b,"\n";` [download] Will print: `2Ö 1Ö` [download] (more or less, depending on PM's escaping mechanisms) -- dakkar - Mobilis in mobile Most of my code is tested... Perl is strongly typed, it just has very few types (Dan)	[reply] [d/l] [select]