Re^6: Character encoding of microns

hi,

So, how would i get round the problem of question marks being both displayed in my terminal for microns and also in any output that is written to a file? When i open my output file in a hex editor, a 3F is displayed for the question mark - indicating that an actual ? is written and it isnt a foreign character. No strange chars like above show up.

Im think im hitting a brick wall with this.

Thanks

Joe

Eschew obfuscation, espouse eludication!

Comment on Re^6: Character encoding of microns

Replies are listed 'Best First'.
Re^7: Character encoding of microns by graff (Chancellor) on Feb 13, 2009 at 06:10 UTC
The ASCII question mark is typically what you get when something tries to convert some unicode character into some non-unicode character set that does not contain the character in question. For example, the following script will produce "foo??", because the string literal has unicode Cyrillic for the fourth and fifth characters, but perl is being told to convert it to iso-8859-1 (Latin-1), which does not contain any Cyrillic characters -- that is, the unicode code points for Cyrillic cannot be mapped into the single-byte character codes for Latin-1, so the conversion produces "?" instead. `perl -MEncode -le '$_=encode("iso-8859-1","foo\x{041d}\x{0418}"); prin +t'` [download] It's not just perl that does this. Anything/everything that supports conversion between unicode and other encodings will behave the same way when faced with the same inappropriate task. To figure out where the question marks are coming from, figure out the last point where the data were in unicode, and what sort of bad assumption is being made at that point to convert the encoding to something else.	[reply] [d/l]

Replies are listed 'Best First'.

Re^7: Character encoding of microns
by graff (Chancellor) on Feb 13, 2009 at 06:10 UTC

For example, the following script will produce "foo??", because the string literal has unicode Cyrillic for the fourth and fifth characters, but perl is being told to convert it to iso-8859-1 (Latin-1), which does not contain any Cyrillic characters -- that is, the unicode code points for Cyrillic cannot be mapped into the single-byte character codes for Latin-1, so the conversion produces "?" instead.

perl -MEncode -le '$_=encode("iso-8859-1","foo\x{041d}\x{0418}"); prin
+t'
[download]

To figure out where the question marks are coming from, figure out the last point where the data were in unicode, and what sort of bad assumption is being made at that point to convert the encoding to something else.

[reply]
[d/l]