in reply to Re: i18n/utf8 problem, 'utf8 "\xF8" does not map to Unicode'
in thread i18n/utf8 problem, 'utf8 "\xF8" does not map to Unicode'
Thanks, Juerd, for your reply.
>How are you READING the UTF-8 data? ... However, if you use :utf8 for input, you're in for trouble (malfunction and security bugs). Always use :encoding for text input.
I think that's what I did -- see the second line of the code:
use open ":encoding(utf8)";>The error message about 0xF8 (which is the Danish ø character, not æ, which is indeed 0xE6) suggests to me that the input is NOT UTF-8, but instead ISO-8859-1 or ISO-8859-15, and the :utf8 was used.
Aha. I checked the file the user sent me:
$ file a.a a.a: ISO-8859 text
My program requires utf8 input, but the user was giving it iso-8859. I think when I cut and pasted it in a utf8-aware editor, it got changed into utf8.
Although my documentation states that the input file has to be utf8, is there any way I can make an explicit check for a bogus encoding? I suppose the crudest thing I could do would be to look at the output of the unix "file" command, but I wonder if there's something more elegant.
|
|---|
| Replies are listed 'Best First'. | |
|---|---|
|
Re^3: i18n/utf8 problem, 'utf8 "\xF8" does not map to Unicode'
by Juerd (Abbot) on Feb 25, 2008 at 10:23 UTC |