mje has asked for the wisdom of the Perl Monks concerning the following question:

Yesterday a colleague passed me some code he had written saying it was not working properly but when he added 'use utf8' it worked. I asked him if his script was written in UTF-8 and he claimed it wasn't and indeed it was not. I could not immediately see why so reduced it to a few lines which demonstrate the problem. I can make it work but I cannot explain why. Any ideas?

#!/usr/bin/perl use strict; use warnings; use fred; my @line; format STDOUT_TOP = @<<<<<< @<<< @>>>>>>>>>>> "a", "b", "c" . format STDOUT = @<<<<<< @<<< @########.## @line . $line[0] = "\x{20ac}"; $line[1] = 'fre'; $line[2] = 2; write;
package fred; use encoding 'utf-8'; 1;
This produces:
a b c € fre < 2.00

He was not getting Wide character in print because he used the fred package and it contains an "encoding utf-8" which adds UTF-8 layer to STDOUT/STDIN. However, where does that '<' come from?

Strangely if you add "use encoding 'utf-8'" to the main script the '<' goes away but much more strange to me is that when he originally passed this to me he claimed adding 'use utf8' to the main script made the '<' go away. I thought all 'use utf8' did was allow you to write your Perl in UTF-8 and yet it appears to be changing the way this script works. Also, moving the 'encoding' down the script to anywhere after the format lines but before the write makes it return to printing '<'

Can anyone explain these strange results.

Replies are listed 'Best First'.
Re: Strange unicode behaviour with UTF8 encoding and format
by moritz (Cardinal) on Feb 18, 2010 at 09:55 UTC

    I can't explain the behaviour you are observing, but I want to comment on a few things nonetheless:

    I usually discourage people from using encoding, because it has many known problems - one of which is related to formats. Also to quote the documentation:

    At any rate, the very use of format is questionable when it comes to unicode characters since you have to consider such things as character width (i.e. double-width for ideographs) and directions (i.e. BIDI for Arabic and Hebrew).

    Instead I write my scripts in UTF-8, and use the utf8 and open pragmas.

      Thanks, the character width issue in format is rather obvious but I'd not noticed the problematic example documented. Personally I've never used encoding and so far I've not had a need to use utf8 in my scripts although I am mostly dealing with unicode data. I was just a little mystified as to what was going on here.