Thank you very much, Tux! Thankfully, one can still get kind and gracious help with Perl problems from Perl experts on PerlMonks.

Upgrading to a bleadperl version of IO isn't practical for me in my environment. So for now, I'll simply not use the open pragma to set default I/O layers and, instead, open files and set I/O layers explicitly in my Perl programs, like this…

#!perl

use strict;
use warnings;

use autodie qw( open close );
use English qw( -no_match_vars );
use Text::CSV_XS;

my $csv = Text::CSV_XS->new({
    always_quote => 1,
    binary       => 1,
    eol          => $INPUT_RECORD_SEPARATOR,
});

binmode STDOUT, ':encoding(UTF-8)';

for my $file (@ARGV) {
    open my $fh, '<:encoding(UTF-8)', $file;

    while (my $fields = $csv->getline($fh)) {
        $csv->print(\*STDOUT, $fields);
    }

    $csv->eof() or $csv->error_diag();

    close $fh;
}

exit 0;
[download]

Now if I could just figure out how best to handle UTF-8 CSV files that have byte order marks in them. ;-) Text::CSV_XS alone chokes on them. I'm currently doing this…

use File::BOM qw( open_bom );

open my $input_fh, '<:via(File::BOM)', $input_file;
open my $output_fh, '>:encoding(UTF-8):via(File::BOM)', $output_file;
[download]

Is this The Right Way?

Let me try to simplify that a bit ...

use Text::CSV_XS;

my $csv = Text::CSV_XS->new ({
    auto_diag    => 1, # Let Text::CSV_XS do the analysis
    always_quote => 1,
    binary       => 1,
    eol          => $INPUT_RECORD_SEPARATOR,
    });

binmode STDOUT, ':encoding(UTF-8)';

for my $file (@ARGV) {
    open my $fh, '<:encoding(UTF-8)', $file;

    while (my $fields = $csv->getline ($fh)) {
        $csv->print (*STDOUT, $fields); # no need for a reference
        }
    # due to auto_diag, no need for error checking here
    close $fh;
    }
[download]

If this script is to sanitize CSV data, I'd advice TWO csv objects. One for parsing, that does not pass the always_quote and eol attribute, and one for output. The advantage is that all legal line-endings are parsed well automatically, even if mixed.

I have no neat way to the BOM problem other than what you already use.


Enjoy, Have FUN! H.Merijn

Thank you, again, Tux. I genuinely appreciate the tips. I'll brush up on auto_diag.

The BOM is a nuisance, especially in CSV files. In one of my real programs that uses Text::CSV_XS (what I posted here is a reduction that simply demonstrates a specific problem I was having), I'm stymied by the confluence of byte order marks in UTF-8 files that force me to use File::BOM and, unfortunately, some malformed UTF-8 text in the data that kills CSV parsing with this unforgiving error message:

utf8 "\xEC" does not map to Unicode at C:/strawberry/perl/lib/Encode.p
+m line 176.
[download]

I don't know how to tell Text::CSV_XS or File::BOM to tell Encode to lighten up already about one or two bogus characters! :-(