I'm trying to get my set of programs to work properly with UTF-8, and I'm running into all kinds of issues. Isn't UTF-8 is the default everywhere now? In debian and Ubuntu at least it has been for a while, but not so in perl it seems. How can I get perl to DWIM: Use UTF-8 everywhere (unless I e.g. open O, ">:raw", $file or die)?

Here are my furstrating experiences and what I've had to do. I'm hoping I'm missing some important use utf8completely; or something that will make my perl and UTF-8 life easier.

First I discover, that even though my source is written in UTF-8 and that is the default on my system, I still need to use utf8; in every file.

If I want files read/written properly, I also need: use open qw{:encoding(utf8) :std}; or the shorter use open ":locale"; in every .pl or .cgi file. But hey, not before a use Foo; if Foo.pm has a non-utf8 character anywhere in the file (even if it is in a comment) or I get a warning caused by use: utf8 "\xA9" does not map to Unicode.. So that means I now have to be careful to use open ... last or at least after use-ing such libs (most of which I admittedly wrote myself). Just confusing that use-ing a lib that doesn't declare use utf8; still needs to be valid utf-8 nonetheless, don't you think?

So far so good

Now Data::Dumper. perl -e 'use utf8; use open qw(:locale); use Data::Dumper; print Dumper("ü")' prints out:

$VAR1 = "\x{fc}";

Ok, so this apparently is not considered a bug in Data::Dumper. But it doesn't DWIM (just give me the ü!!!) I'm writing my code in utf8, so 'ü' is much more handy than "\x{fc}" even if they are equivalent behind the scenes. Especially now that I'm trying to use Data::Dumper to debug my UTF-8 issues. But AHA! $Data::Dumper::Useperl=1 to the rescue: perl -e 'use utf8; use open qw(:locale); use Data::Dumper; $Data::Dumper::Useperl= 1; print Dumper("ü")' prints out:

$VAR1 = 'ü';

Fantastic even though it is much slower. Now I can trust the output of Data::Dumper (even though a little doubt remains: Why does Useperl=1 produce different output? Will this change in future versions of perl? Oh, never mind...)! Oh, wait; now I have problems with:

• Log::Log4perl (I needed a line like log4perl.appender.name.utf8=1 in my config file - didn't default to accepting utf-8)
• Term::ReadLine(::Gnu) I still haven't figured out why the utf8::decode($str) is neccessary here. I thought I told perl that all input and output was to be in utf8 with the use open... line.

  #!/usr/bin/perl -w
  use strict;
  use utf8;
  use Data::Dumper;
  use Encode;
  $Data::Dumper::Useperl=1;
  use Term::ReadLine;
  use open (':locale');
  my $term = Term::ReadLine->new('xmlapish');
  my $str = $term->readline('> ');
  
  # Need one of these or File size below becomes 4. 4!!! The 4 bytes are WRONG in
  # any way you choose to look at it!
  #
  # use Encode; $str = decode_utf8($str);
  utf8::decode($str);
  
  chomp $str;
  print Dumper($str);
  
  open O, ">", "file" or die;
  print O $str;
  close O;
  printf "File size: %d\n", [stat('file')]->[7];
  

  Googling for "Term::ReadLine utf8" or "GNU ReadLine utf8" both give nothing interesting.

And the ('Data::Dumper', 'Log::Log4perl', 'Term::ReadLine') list was just from the ½ hour testing I did after I tried to switch from ISO-8859-1 to UTF-8. There are probably many more UTF-8 issues waiting to be found. I really find it a frustrating experience to use perl and UTF-8 togther, I must say! Am I doing something fundamentally wrong?