Lightknight has asked for the wisdom of the Perl Monks concerning the following question:
I'm trying to get my set of programs to work properly with UTF-8, and I'm running into all kinds of issues. Isn't UTF-8 is the default everywhere now? In debian and Ubuntu at least it has been for a while, but not so in perl it seems. How can I get perl to DWIM: Use UTF-8 everywhere (unless I e.g. open O, ">:raw", $file or die)?
Here are my furstrating experiences and what I've had to do. I'm hoping I'm missing some important use utf8completely; or something that will make my perl and UTF-8 life easier.
First I discover, that even though my source is written in UTF-8 and that is the default on my system, I still need to use utf8; in every file.
If I want files read/written properly, I also need: use open qw{:encoding(utf8) :std}; or the shorter use open ":locale"; in every .pl or .cgi file. But hey, not before a use Foo; if Foo.pm has a non-utf8 character anywhere in the file (even if it is in a comment) or I get a warning caused by use: utf8 "\xA9" does not map to Unicode.. So that means I now have to be careful to use open ... last or at least after use-ing such libs (most of which I admittedly wrote myself). Just confusing that use-ing a lib that doesn't declare use utf8; still needs to be valid utf-8 nonetheless, don't you think?
So far so good
Now Data::Dumper. perl -e 'use utf8; use open qw(:locale); use Data::Dumper; print Dumper("ü")' prints out:
$VAR1 = "\x{fc}";
Ok, so this apparently is not considered a bug in Data::Dumper. But it doesn't DWIM (just give me the ü!!!) I'm writing my code in utf8, so 'ü' is much more handy than "\x{fc}" even if they are equivalent behind the scenes. Especially now that I'm trying to use Data::Dumper to debug my UTF-8 issues. But AHA! $Data::Dumper::Useperl=1 to the rescue: perl -e 'use utf8; use open qw(:locale); use Data::Dumper; $Data::Dumper::Useperl= 1; print Dumper("ü")' prints out:
$VAR1 = 'ü';
Fantastic even though it is much slower. Now I can trust the output of Data::Dumper (even though a little doubt remains: Why does Useperl=1 produce different output? Will this change in future versions of perl? Oh, never mind...)! Oh, wait; now I have problems with:
- Log::Log4perl (I needed a line like log4perl.appender.name.utf8=1 in my config file - didn't default to accepting utf-8)
- Term::ReadLine(::Gnu) I still haven't figured out why the utf8::decode($str) is neccessary here. I thought I told perl that all input and output was to be in utf8 with the use open... line.
#!/usr/bin/perl -w use strict; use utf8; use Data::Dumper; use Encode; $Data::Dumper::Useperl=1; use Term::ReadLine; use open (':locale'); my $term = Term::ReadLine->new('xmlapish'); my $str = $term->readline('> '); # Need one of these or File size below becomes 4. 4!!! The 4 bytes are WRONG in # any way you choose to look at it! # # use Encode; $str = decode_utf8($str); utf8::decode($str); chomp $str; print Dumper($str); open O, ">", "file" or die; print O $str; close O; printf "File size: %d\n", [stat('file')]->[7];
And the ('Data::Dumper', 'Log::Log4perl', 'Term::ReadLine') list was just from the ½ hour testing I did after I tried to switch from ISO-8859-1 to UTF-8. There are probably many more UTF-8 issues waiting to be found. I really find it a frustrating experience to use perl and UTF-8 togther, I must say! Am I doing something fundamentally wrong?
|
---|
Replies are listed 'Best First'. | |
---|---|
Re: Why am I having so much trouble and pain with UTF-8 in perl?
by moritz (Cardinal) on Oct 01, 2010 at 15:14 UTC | |
by Lightknight (Beadle) on Jun 25, 2015 at 10:19 UTC |