in reply to Re: i18n/utf8 problem, 'utf8 "\xF8" does not map to Unicode'
in thread i18n/utf8 problem, 'utf8 "\xF8" does not map to Unicode'

Thanks, Juerd, for your reply.

>How are you READING the UTF-8 data? ... However, if you use :utf8 for input, you're in for trouble (malfunction and security bugs). Always use :encoding for text input.

I think that's what I did -- see the second line of the code:

use open ":encoding(utf8)";

>The error message about 0xF8 (which is the Danish ø character, not æ, which is indeed 0xE6) suggests to me that the input is NOT UTF-8, but instead ISO-8859-1 or ISO-8859-15, and the :utf8 was used.

Aha. I checked the file the user sent me:

$ file a.a a.a: ISO-8859 text

My program requires utf8 input, but the user was giving it iso-8859. I think when I cut and pasted it in a utf8-aware editor, it got changed into utf8.

Although my documentation states that the input file has to be utf8, is there any way I can make an explicit check for a bogus encoding? I suppose the crudest thing I could do would be to look at the output of the unix "file" command, but I wonder if there's something more elegant.

Replies are listed 'Best First'.
Re^3: i18n/utf8 problem, 'utf8 "\xF8" does not map to Unicode'
by Juerd (Abbot) on Feb 25, 2008 at 10:23 UTC

    use open ":encoding(utf8)";

    Good.

    My program requires utf8 input, but the user was giving it iso-8859.

    There's no easy way to fix the user. :)

    Juerd # { site => 'juerd.nl', do_not_use => 'spamtrap', perl6_server => 'feather' }