perlquestion
Lightknight
<p>I'm trying to get my set of programs to work properly with UTF-8, and I'm running into all kinds of issues. Isn't UTF-8 is the default everywhere now? In debian and Ubuntu at least it has been for a while, but not so in perl it seems. How can I get perl to DWIM: Use UTF-8 <strong>everywhere</strong> (unless I e.g. <code>open O, ">:raw", $file or die</code>)?</p>
<p>Here are my furstrating experiences and what I've had to do. I'm hoping I'm missing some important <code>use utf8completely;</code> or something that will make my perl and UTF-8 life easier.</p>
<p>First I discover, that even though my source is written in UTF-8 and that is the default on my system, I still need to <code>use utf8;</code> in every file.</p>
<p>If I want files read/written properly, I also need: <code>use open qw{:encoding(utf8) :std};</code> or the shorter <code>use open ":locale";</code> in every .pl or .cgi file. But hey, not before a <code>use Foo;</code> if Foo.pm has a non-utf8 character anywhere in the file (even if it is in a comment) or I get a warning caused by <code>use</code>: <code>utf8 "\xA9" does not map to Unicode.</code>. So that means I now have to be careful to <code>use open ...</code> <em>last</em> or at least after use-ing such libs (most of which <em>I</em> admittedly wrote myself). Just confusing that use-ing a lib that doesn't declare <code>use utf8;</code> still needs to be valid utf-8 nonetheless, don't you think?</p>
<p>So far so good</p>
<p>Now Data::Dumper. <code>perl -e 'use utf8; use open qw(:locale); use Data::Dumper; print Dumper("ü")'</code> prints out:</p>
<pre>
$VAR1 = "\x{fc}";
</pre>
<p>Ok, so this apparently is <a href="https://rt.cpan.org/Public/Bug/Display.html?id=28607">not considered a bug in Data::Dumper</a>. But it doesn't DWIM (just give me the ü!!!) I'm writing my code in utf8, so 'ü' is much more handy than "\x{fc}" even if they are equivalent behind the scenes. Especially now that I'm trying to use Data::Dumper to debug my UTF-8 issues. But AHA! $Data::Dumper::Useperl=1 to the rescue: <code>perl -e 'use utf8; use open qw(:locale); use Data::Dumper; $Data::Dumper::Useperl= 1; print Dumper("ü")'</code> prints out:</p>
<pre>
$VAR1 = 'ü';
</pre>
<p>Fantastic even though it is much slower. Now I can trust the output of Data::Dumper (even though a little doubt remains: Why does Useperl=1 produce different output? Will this change in future versions of perl? Oh, never mind...)! Oh, wait; now I have problems with:</p>
<ul>
<li><em>Log::Log4perl</em> (I needed a line like <code>log4perl.appender.name.utf8=1</code> in my config file - didn't default to accepting utf-8)</li>
<li><em>Term::ReadLine(::Gnu)</em> I still haven't figured out why the <code>utf8::decode($str)</code> is neccessary here. I thought I told perl that all input and output was to be in utf8 with the <code>use open...</code> line.
<readmore title="See the code snippet">
<pre>
#!/usr/bin/perl -w
use strict;
use utf8;
use Data::Dumper;
use Encode;
$Data::Dumper::Useperl=1;
use Term::ReadLine;
use open (':locale');
my $term = Term::ReadLine->new('xmlapish');
my $str = $term->readline('> ');
# Need one of these or File size below becomes 4. 4!!! The 4 bytes are WRONG in
# any way you choose to look at it!
#
# use Encode; $str = decode_utf8($str);
utf8::decode($str);
chomp $str;
print Dumper($str);
open O, ">", "file" or die;
print O $str;
close O;
printf "File size: %d\n", [stat('file')]->[7];
</pre>
</readmore>
Googling for "Term::ReadLine utf8" or "GNU ReadLine utf8" both give nothing interesting.
</li>
</ul>
<p>And the ('Data::Dumper', 'Log::Log4perl', 'Term::ReadLine') list was just from the ½ hour testing I did after I tried to switch from ISO-8859-1 to UTF-8. There are probably many more UTF-8 issues waiting to be found.
I really find it a frustrating experience to use perl and UTF-8 togther, I must say! Am I doing something fundamentally wrong?</p>