open(my $data, '<:encoding(UCS-2le)', $file)
[download]

Actually, due to problems with the placement of the :crlf layer,

open(my $data, '<:raw:encoding(UCS-2le):crlf:utf8', $file)
[download]

And the funny char is the BOM (U+FEFF).

read($data, my $bom='', 1);  # Discard BOM.
[download]

(To the best of my knowledge, what Microsoft uses is not really UTF-16 but UCS-2le.)

(To the best of my knowledge, what Microsoft uses is not really UTF-16 but UCS-2le.)

I used to think that UCS-2LE was a synonym for UTF-16LE (likewise with BE instead of LE). But I then found this bit in a FAQ at unicode.org:

Q: What is the difference between UCS-2 and UTF-16?

A: UCS-2 is what a Unicode implementation was up to Unicode 1.1, before surrogate code points and UTF-16 were added as concepts to Version 2.0 of the standard. This term should be now be avoided.

When interpreting what people have meant by "UCS-2" in past usage, it is best thought of as not a data format, but as an indication that an implementation does not interpret any supplementary characters. In particular, for the purposes of data exchange, UCS-2 and UTF-16 are identical formats. Both are 16-bit, and have exactly the same code unit representation.

The effective difference between UCS-2 and UTF-16 lies at a different level, when one is interpreting a sequence code units as code points or as characters. In that case, a UCS-2 implementation would not handle processing like character properties, codepoint boundaries, collation, etc. for supplementary characters.

The current version of Unicode is 5.something, so there's not much point in using the "UCS-2" terminology these days.

And yes, Microsoft tends to favor the LE byte order, especially for text data (MS-Word, and "plain-text" exports from MS-Office products). But cells in Excel spreadsheets are, for some reason, stored as BE.

As far as perl encoding layers are concerned, UTF-16 (with no byte-order spec) tends to mean: for output, byte order is determined by the cpu apparently BE by default; for input, byte order is determined by a stream-initial BOM (if the BOM isn't there, perl complains about it; if it is there, perl does not will remove it for you).

(updated last paragraph to reflect ikegami's corrections -- thanks, ike)

To me, there are two important differences between UCS-2 and UTF-16.

The first important difference is that UCS-2 can only represent U+0000 to U+FFFF, whereas UTF-16 can represent any UNICODE character.

The second important difference is the number of bytes UCS-2 and UTF-16 use to store a character. Each UCS-2 character is exactly 16 bits in size, whereas UTF-16 is like UTF-8. Some characters require more than one word.

for output, byte order is determined by the cpu

No. I'm on an x86 (LE machine), but UTF-16be was used.

for input, byte order is determined by a stream-initial BOM (if the BOM isn't there, perl complains about it; if it is there, perl does not remove it for you).

No. Perl *does* remove it for you, just like it adds it for you for output.

You could instead try the largely compatible Text::CSV::Unicode.

#!/usr/bin/perl
use warnings; use strict;

# solution 2, converted to Text::CSV::Unicode

# replacement for Text::CSV_XS
use Text::CSV::Unicode;

my $csv = Text::CSV::Unicode->new();
my $count=0;

my $file = 'PS0002_9_2006b.txt';
if (defined $ARGV[0]) {
    $file = $ARGV[0];
}

my $sum = 0;
# ikegami's fix
open(my $data, '<:raw:encoding(UCS-2le):crlf:utf8', $file) or die "Cou
+ld not open '$file'\n";
while (my $line = <$data>) {
    chomp $line;
    exit if ($count >= 20);
    $count++;
    if ($csv->parse($line)) {
        my @columns = $csv->fields();

        $sum += $columns[2];
    } else {
        warn "Line could not be parsed: $line\n";
    }
}
print "$sum\n";
# end solution 2_______________________
[download]

Text::CSV might soon be extended with a layer that deals with encodings, somewhat like this:

use Text::CSV::Encoded;
my $csv = Text::CSV::Encoded->new ({
    encoding     => "utf-8",      # Both in and out
    encoding_in  => "utf-16le",   # Only the input
    encoding_out => "cp1252",     # Only the output
    });
[download]

Until then, I think

binmode STDOUT, ":utf8";
my $csv = Text::CSV_XS->new ({ binary => 1 });
open my $fh, "<:encoding(utf-16le)", $file or die "$file: $!";
while (my $row = $csv->getline ($fh)) {
    print $row->[4];
    }
[download]

should work


Enjoy, Have FUN! H.Merijn

It's already been covered that it should be

open my $fh, "<:raw:encoding(utf-16):crlf:utf8", $file
   or die "$file: $!";
[download]

or more precisely,

open my $fh, "<:raw:encoding(ucs-2le):crlf:utf8", $file
   or die "$file: $!";
read($fh, my $bom='', 1);
[download]

And no, it doesn't work. Not if the data contains any non-ASCII characters, at least, but that's the whole point of this exercise. The UTF8 flag gets turned off, so the UTF-8 encoding of the characters is treated as iso-latin-1.

For example, if a field contains <"é">, Text::CSV_XS returns the two characters <é> instead of <é>. (I'm using angled brackets to quote to avoid confusion with the double-quotes in the CSV file.)

For example, if a field contains <"♠">, Text::CSV_XS returns the three characters <♣> instead of <♠>.

The flag needs to be reinstated, so it should be:

use Encode qw( _utf8_on );

my $csv = Text::CSV_XS->new ({ binary => 1 });

# UTF-16 or UCS-2 file with BOM and CRLF or LF line endings.
open my $fh, "<:raw:encoding(utf-16):crlf:utf8", $file
   or die "$file: $!";

while (my $row = $csv->getline ($fh)) {
    # Fix inability of CSV_XS to handle UTF8 strings.
    _utf8_on($_) for @$row;

    print $row->[4];
}
[download]

There is at least one other problem with treating characters encoded using UTF-8 no differently then characters encoded using iso-latin-1 as Text::CSV_XS does.

If any of eol, sep_char, etc is passed a string with the UTF8 flag off and it contains a character in [\x80-\xFF], Text::CSV_XS can generate false positives. However, this is unlikely to ever happen.

Text::CSV might soon be extended with a layer that deals with encodings

I don't see the point, since Text::CSV doesn't open any file handles. All it needs to do is respect the UTF8 flag on strings it receives via getline, eol, sep_char, etc. Currently (well, 0.34 and presumably 0.45), it ignores it.

Could you give it a go with Text::CSV_XS 0.50, which I just uploaded? It now promotes the UTF-8 setting you got from the file-handle encoding to the fields in getline () and parse ()