Re: libXSLT and  

I don't understand the previous two replies, since character 160 has the same meaning in Latin-1 as it does in Unicode. Namely, none. That is a C1 control code with no defined meaning.

In the Windows 1252 character set, the C1 zone has extra characters in it, making it a super-set of Latin 1. I guess 160 (0xA0) is the non-breaking space character.

So, the code appears to be converting from a character set where 0xa0 means A-hat, which is 194 in Latin-1.

Hmm, perhaps the resulting UTF-8 is being treated as two characters later. A0 will be encoded as C3 80, which if re-displayed as Windows 1252 is A-tilde then the wanted non-breaking space. If your note was a typo (tilde instead of angle hat), that's what's happening.

Comment on Re: libXSLT and

Replies are listed 'Best First'.
Re: Re: libXSLT and   by grantm (Parson) on Dec 18, 2002 at 20:12 UTC
There's a subtle distinction going on here. Character 160 (0xA0) is a non-breaking space in both Latin-1 and Unicode. However the binary representation of a Unicode character depends on what encoding you use: if you encode Unicode with UTF-16 then every character will be represented as two bytes and character 160 will be 0x00 0xA0 (or 0xA0 0x00 depending on endian-ness) UTF-8 on the other hand uses a variable number of bytes (from 1 to 7) to represent unicode characters. All characters from 0-127 are a single byte (so plain ASCII files are also UTF-8) and all characters from 128-65535 (and beyond?) are two or more bytes. Note also, that the Windows 1252 character set is not a super-set of Latin-1. Latin-1 is an 8 bit character set in which every character position has been assigned. Microsoft decided the the control characters in the 0x80-0x9F range were not useful so in CP1252 they removed those character assignments and replaced them with other characters (such as 'smart quotes'). I think we'd all agree that 7-bit ASCII is insufficient for general use outside of the US (in fact it can't even represent the US 'cent' symbol: ¢). The 8-bit Latin-1 character set is a 'point solution' for western european nations but it is no longer sufficient for them either (eg: no Euro symbol: €). The clear way forward is widespread adoption of Unicode - it's here now and it works already. Clinging to and tweaking region-specific 8 bit encodings is a dead-end strategy. Sorry if this sounds like a rant - it's not intended to be personal. People just seem to waste a lot of time trying to translate perfectly good Unicode characters back into 'legacy' encodings instead of just using them as they are.	[reply]

Replies are listed 'Best First'.

Re: Re: libXSLT and  
by grantm (Parson) on Dec 18, 2002 at 20:12 UTC

There's a subtle distinction going on here. Character 160 (0xA0) is a non-breaking space in both Latin-1 and Unicode. However the binary representation of a Unicode character depends on what encoding you use:

if you encode Unicode with UTF-16 then every character will be represented as two bytes and character 160 will be 0x00 0xA0 (or 0xA0 0x00 depending on endian-ness)
UTF-8 on the other hand uses a variable number of bytes (from 1 to 7) to represent unicode characters. All characters from 0-127 are a single byte (so plain ASCII files are also UTF-8) and all characters from 128-65535 (and beyond?) are two or more bytes.

Note also, that the Windows 1252 character set is not a super-set of Latin-1. Latin-1 is an 8 bit character set in which every character position has been assigned. Microsoft decided the the control characters in the 0x80-0x9F range were not useful so in CP1252 they removed those character assignments and replaced them with other characters (such as 'smart quotes').

I think we'd all agree that 7-bit ASCII is insufficient for general use outside of the US (in fact it can't even represent the US 'cent' symbol: ¢). The 8-bit Latin-1 character set is a 'point solution' for western european nations but it is no longer sufficient for them either (eg: no Euro symbol: €). The clear way forward is widespread adoption of Unicode - it's here now and it works already. Clinging to and tweaking region-specific 8 bit encodings is a dead-end strategy.

Sorry if this sounds like a rant - it's not intended to be personal. People just seem to waste a lot of time trying to translate perfectly good Unicode characters back into 'legacy' encodings instead of just using them as they are.

[reply]