rwadkins has asked for the wisdom of the Perl Monks concerning the following question:

I have a string of Japanese text in utf-8 format (I've looked at the bytes and they seem to be correctly addressing the utf-8 code pages). Some of the characters are 3 bytes long (e4 bb 96 - 他).

Encode::decode complains about the wide characters. However, Encode::decode_utf8 doesn't. Using the debugger typing 'x $var' outputs the string to the screen properly (although my terminal can't display the 3 byte characters, it replaces them with boxes).

However, using warn or print causes the string to be mangled. It's also getting mangled when it goes to the database. I'm running perl with -CS, and I've tried setting binmode to utf8 for both STDOUT and STDERR.

I feel like I'm missing something fundamental about these strings, but I keep having to relearn about utf8 encoding every time I run into encoding problems.

Update

Turns out I hadn't looked at the strong close enough. There were \u200b characters in the string that Encode didn't like. Stripping them out resolved the issue.

Replies are listed 'Best First'.
Re: Triple Byte UTF8 Characters
by ikegami (Patriarch) on Mar 29, 2011 at 16:08 UTC

    Decoding is the process of taking a string of bytes and returning the string of characters those bytes represent.

    If you're getting that warning, it means you're passing something other than bytes to decode. It could mean you are decoding a string you already decoded, for example. 

    use strict;
use warnings;
use feature qw( say );

use open ':std', ':encoding(UTF-8)';

use Encode qw( decode );

$_ = "\xe4\xbb\x96";
say length;

$_ = decode("UTF-8", $_);
say length;

$_ = decode("UTF-8", $_);  # XXX Bug
say length;

    [download]
    3
1
Cannot decode string with wide characters at [...].

    [download]
[reply]
[d/l]
[select]
Re: Triple Byte UTF8 Characters
by moritz (Cardinal) on Mar 29, 2011 at 17:55 UTC
    Here's a working example under linux: 
    # test.pl
use strict;
use warnings;
binmode $_, ':encoding(UTF-8)' for *STDIN, *STDOUT, *STDERR;
my $line = <>;
print $line;
chomp $line;
print length($line), "\n";
__END__

# echo -e "\xe4\xbb\x96"|perl test.pl

    [download]

    Output:

    他
    1

    The hard thing is that you have to manually keep track of which strings are decoded, and which not. Which is a huge cognitive load, unless you settle for decoding everything through an input IO layer, and encoding everything through an output IO layer.

    Most database drivers have an option to do that for you too.

    See also: Encodings and Unicode.

    Perl 6 - second systems done right
[reply]
[d/l]

Back to Seekers of Perl Wisdom