Text::CSV_XS doesn't do anything with encodings internally, and reads bytes, which is also the reason why EBCDIC is so damn hard to implement in the current structure.

That said, the user is resonsible for the encoding/decoding of the data/fields, as CSV files have no way of telling that to the parser.

_utf8_on ($_) is NOT the way to go. Please read Unicode advice and Perl Unicode totorial for the reasons.

while (my $row = $csv->getline ($fh)) {
    # Fix inability of CSV_XS to handle UTF8 strings.
    utf8::decode ($_) for @$row;

    print $row->[4];
    }

[download]

As a proof of concept, I tried something more simple in the example below

#!/pro/bin/perl

use strict;
use warnings;

use Text::CSV_XS;
use Encode qw( encode decode );

my $csv = Text::CSV_XS->new ({ binary => 1 });

my $file = "test.csv";

open my $fh, ">:encoding(utf-16)", $file or die "$file: $!";
print $fh join ",",
    "\x{0073}e\x{00f1}\x{00f3}\x{0159}",
    123,
    "\x{00c5}\x{0142}\x{00e9}\x{0161}\x{0171}\x{0146}",
    "\r\n";
close $fh;

binmode STDOUT, ":utf8";

open $fh, "<:raw:encoding(utf-16)", $file or die "$file: $!";
while (my $row = $csv->getline ($fh)) {
    print join "," => @$row, "\n";

    utf8::decode ($_) for @$row;
    print join "," => @$row, "\n";
    }

[download]

To show that test.csv now has a BOM:

$ od -t x1 test.csv
0000000 fe ff 00 73 00 65 00 f1 00 f3 01 59 00 2c 00 31
0000020 00 32 00 33 00 2c 00 c5 01 42 00 e9 01 61 01 71
0000040 01 46 00 2c 00 0d 00 0a
0000050

[download]

And the cript output was:

señóÅ,123,ÃÅéšűÅ,,
señóř,123,Åłéšűņ,,

The second line was exactly what I was expecting.


Enjoy, Have FUN! H.Merijn

In reply to Re^4: CSV nightmare by Tux
in thread CSV nightmare by lorenzov

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.