If you were using Perl 5.8, I'd suggest pushing an encoding layer when you opened the file (or after with binmode). As you're not, I won't.

Here's a quick script that reads a file line-by-line and uses pack to set the UTF-8 flag on each string read in. After that flag is set, character semantics work as expected for wide characters that were read in from the file:

  use utf8;
  use CGI::Carp qw(fatalsToBrowser);

  print "Content-type: text/html; charset=utf-8\n\n";

  open(FILE, "<", "/path/to/utf8/file.txt") || die "$!";

  print "<pre>\n";
  while(<FILE>) {
    chomp;
    $_ = set_utf($_);
    my $len = length($_);    # count of chars not bytes
    print "$_", ' ' x (72 - $len), "|\n";
  }
  print "</pre>\n";

  sub set_utf {
    return pack "U0a*", join '', @_;
  }
[download]

I fashioned the script as a CGI script so that you can view the output in your browser - which understands UTF-8 characters (whereas your TTY might not). Given a UTF-8 text file with lines less than 80 characters, this should pad each line out to 80 characters with spaces and then append a '|'. If character semantics are not in force, the length will count bytes rather than characters and the '|'s won't line up.

 open(FILE, "<", "/path/to/utf8/file.txt") || die "$!";

Why don't use perl 5.8.0 and simply specify IO 'layer' when using three-argument form of open?

open(FILE, '<:utf8', '/path/to/utf8/file.txt') || die $!;

Why don't use perl 5.8.0...

um, because the original poster was looking for a 5.6.1 solution. (And if you actually read my reply you'd see that I suggested exactly what you propose in the first line!)

I spent a while reading the unicode documentation with perl and it looks like you need to apply unicode attributes to the filehandle. My 5.6.1 manual only mentions it as being difficult and never actually documents how to make this work. Stepping up to 5.8.0 documentation results in the following gems: binmode DATA, ':utf8 for already opened handles, open(my $fh, '<:utf8', 'anything') for new files. open can be overridden to have unicode semantics by default by using use open ':utf8';. You can read this yourself in the perluniintro document. Perhaps someone here can fill in what the mystery 5.6.1 incantation is.

__SIG__
use B;
printf "You are here %08x\n", unpack "L!", unpack "P4", pack
  "L!", B::svref_2object(sub{})->OUTSIDE;
[download]

I would expect this to be totally revised, when Perl 6 comes out, otherwise I feel some real worry here.

I'm not quite sure what you're getting at here. You will always need to tell Perl that you want it to use UTF-8 encoding when you read a specific file. Sure in the future some of the region-specific encodings such as Latin-1 might lose popularity to Unicode. But if Perl assumed every file was a UTF-8 character stream then Perl would no longer be able to read binary byte streams (or even UTF-16 encoded).

The XML spec provides a way for a program to unambiguously determine the encoding of an XML document. In the absense of this type of in-band information in other text file formats, you will need to specify an encoding.

As you point out, 5.8 provides the very powerful IO layer model for dealing with this and other problems. I don't expect IO layers to disappear in 6.0. And for people stuck with 5.6, pack hack's do provide a workaround.

What is expected to change in the future is that Perl will assume your script itself is UTF-8 encoded. Assuming you use a UTF-8 aware editor, that will allow you to include non-ASCII characters in string literals simply by typing them. At the moment if you want to do that you have to say 'use utf8' in the future that will be assumed (and to quote the docs, "'use utf8' will become a noop").

"You will always need to tell Perl that you want it to use UTF-8 encoding" Unless that's the default. Of course, if you want to read a binary file, you can always ask for that, but there's no reason in this day and age that UTF8 encoding wouldn't be the default.