in reply to Parsing UTF-16LE CSV Records Using Text::CSV*

I would strongly suggest that you convert the utf-16 data to perl-internal utf8 before doing anything else with it. You can convert it back to utf-16 again for output if necessary, but there's no reason to use utf-16 encoding for character-based processing.

Just set the encoding layer on the input file handle to ":encoding(UTF-16LE)" and perl will convert the data to utf8 as it reads from the file. Do the same for the output file handle.

Replies are listed 'Best First'.
Re^2: Parsing UTF-16LE CSV Records Using Text::CSV*
by ikegami (Patriarch) on Jul 20, 2009 at 07:35 UTC

    According to the documentation, that won't work. Both the escape char and the quote char are limited to a single-byte character. U+00FE and U+00FE are not single byte characters in "perl-internal utf8".

    Boo for equating the string's internal encoding with whether it's been decoded or not. See Re: Decoding, Encoding string, how to? (internal encoding).

[reply]
      Boo for equating the string's internal encoding with whether it's been decoded or not.

      If you "decode()" a non-ascii, non-utf8 string (or if it passes through a decoding IO layer on input), and the operation is successful, the string value returned by decode() has the utf8 flag on, and you get character semantics (not byte semantics) when doing stuff with that string -- that's the point of using "decode()" and the encoding IO layer, and that's all I was talking about in my suggestion. (My reply may well have been less than fully helpful for other reasons.)

      As for U+00FE, perhaps I'm just behind the times, not having taken time to explore all the details of 5.10 yet. In 5.8.8, the "perl-internal utf8" storage of characters in the rang 0x80-0xFF is single-byte. They would be converted to multi-byte on output to a utf8-mode file handle. I don't recall at the moment what particular operations are sensitive to (or would reveal) this distinction, but it's there.

[reply]

        and you get character semantics (not byte semantics) when doing stuff with that string

        There's no such thing. If an operation behaves differently depending on the internal encoding of the string, it's a bug. These are being fixed. e.g. pack was fixed in 5.10.0. Regex matches and other are being fixed for 5.12. Text::CSV_XS was fixed in 0.46.

        that's the point of using "decode()" and the encoding IO layer

        Not at all. The point of decode is to decode characters. It has nothing to do with the internal storage of strings.

        You can have decoded characters with the utf8 flag off.
        You can encoded characters with the utf8 flag on.

        If you need to play with the internal encoding, utf8::upgrade and utf8::downgrade are the appropriate tools.

        This is what the previously linked document shows.

        the "perl-internal utf8" storage of characters in the rang 0x80-0xFF is single-byte.

        Impossible. The high bit indicates the presence of a multiple byte char. 

        $ perl -MEncode -le'print length encode "utf8", decode "UTF-16le", "\x
+FE\x00"'
2

        [download]
        or 
        $ perl -MDevel::Peek -MEncode -le'Dump decode "UTF-16le", "\xFE\x00"'
...
  PV = 0x8172e78 "\303\276"\0 [UTF8 "\x{fe}"]
  CUR = 2
...

        [download]

        U+000000-U+00007F: One byte
        U+000080-U+0007FF: Two bytes
        U+000800-U+00FFFF: Three bytes
        U+010000-U+10FFFF: Four bytes

[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom