in reply to Re^4: Speeds vs functionality
in thread Speeds vs functionality

Text::CSV_PP is able to parse that text, at least in UTF-8. 

use v5.12;
use warnings;
use utf8::all;
use Text::CSV_PP;

my $csv = Text::CSV_PP->new ( 
    { binary      => 1 , 
      quote_char  => '🎥' ,
      escape_char => '🎥' ,
      sep_char    => '🎬'  } )
  or die "Cannot use CSV_PP: "
   .Text::CSV_PP->error_diag ();

my @rows;
my $fh = *DATA;
while ( my $row = $csv->getline( $fh ) ) {
         push @rows, $row;
}
$csv->eof or $csv->error_diag();
for ( @rows ) {
    printf("%-25s%s\n", $_->[0], $_->[4]);
}
__DATA__
🎥Film🎥🎬🎥Year🎥🎬🎥Awards🎥🎬🎥Nominations🎥🎬🎥Director🎥
🎥12 Years a Slave🎥🎬2013🎬3🎬9🎬🎥🎥🎥 Steve McQueen🎥
🎥Argo🎥🎬2012🎬3🎬7🎬🎥🎥🎥 Ben Affleck🎥
🎥The Artist🎥🎬2012🎬5🎬10🎬🎥🎥🎥 Michel Hazanavicius🎥
🎥The King's Speech🎥🎬2010🎬4🎬12🎬🎥🎥🎥 Tom Hooper🎥
🎥The Hurt Locker🎥🎬2009🎬6🎬9🎬🎥🎥🎥 Kathryn Bigelow🎥
🎥Slumdog Millionaire🎥🎬2008🎬8🎬10🎬🎥🎥🎥 Danny Boyle🎥
🎥No Country for Old Men🎥🎬2007🎬4🎬8🎬🎥🎥🎥 Joel Coen 🎥🎥 Ethan Coen🎥
🎥The Departed🎥🎬2006🎬4🎬5🎬🎥🎥🎥 Martin Scorsese🎥

Output: 

Film                     Director
12 Years a Slave         🎥 Steve McQueen
Argo                     🎥 Ben Affleck
The Artist               🎥 Michel Hazanavicius
The King's Speech        🎥 Tom Hooper
The Hurt Locker          🎥 Kathryn Bigelow
Slumdog Millionaire      🎥 Danny Boyle
No Country for Old Men   🎥 Joel Coen 🎥 Ethan Coen
The Departed             🎥 Martin Scorsese

Replies are listed 'Best First'.
Re^6: Speeds vs functionality
by Jim (Curate) on Aug 01, 2014 at 01:38 UTC

    Booyah! farang FTW!

    Here's my test with a very lightly refactored version of the same script:

    use v5.14;
use strict;
use warnings;
use utf8;

use Text::CSV_PP;

binmode STDOUT, ':encoding(UTF-8)';

my $csv = Text::CSV_PP->new({
    sep_char    => '🎬',
    quote_char  => '🎥',
    escape_char => '🎥',
    binary      => 1,
});

my @rows;

my $fh = *DATA;

while (my $row = $csv->getline($fh)) {
    push @rows, $row;
}

$csv->eof() or $csv->error_diag();

for my $row (@rows) {
    $row->[4] =~ s/\n\s*/, /g;

    printf "%-24s %s\n", $row->[0], $row->[4];
}

exit 0;

__DATA__
🎥Film🎥🎬🎥Year🎥🎬🎥Awards🎥🎬🎥Nominations🎥🎬🎥Director🎥
🎥12 Years a Slave🎥🎬2013🎬3🎬9🎬🎥🎥🎥 Steve McQueen🎥
🎥Argo🎥🎬2012🎬3🎬7🎬🎥🎥🎥 Ben Affleck🎥
🎥The Artist🎥🎬2012🎬5🎬10🎬🎥🎥🎥 Michel Hazanavicius🎥
🎥The King's Speech🎥🎬2010🎬4🎬12🎬🎥🎥🎥 Tom Hooper🎥
🎥The Hurt Locker🎥🎬2009🎬6🎬9🎬🎥🎥🎥 Kathryn Bigelow🎥
🎥Slumdog Millionaire🎥🎬2008🎬8🎬10🎬🎥🎥🎥 Danny Boyle🎥
🎥No Country for Old Men🎥🎬2007🎬4🎬8🎬🎥🎥🎥 Joel Coen
🎥🎥 Ethan Coen🎥
🎥The Departed🎥🎬2006🎬4🎬5🎬🎥🎥🎥 Martin Scorsese🎥

    This correctly produces:

    Film                     Director
12 Years a Slave         🎥 Steve McQueen
Argo                     🎥 Ben Affleck
The Artist               🎥 Michel Hazanavicius
The King's Speech        🎥 Tom Hooper
The Hurt Locker          🎥 Kathryn Bigelow
Slumdog Millionaire      🎥 Danny Boyle
No Country for Old Men   🎥 Joel Coen, 🎥 Ethan Coen
The Departed             🎥 Martin Scorsese

    Notice that this version handles the literal newline (\n, CR-LF) in the Coen brothers record, which I change to ',' in the output.

    Thank you, farang. I stand corrected:  there is a Unicode-capable CSV parser/generator Perl module on CPAN. And I think you just solved a very long-lived problem for me.

[reply]
[d/l]
[select]
Re^6: Speeds vs functionality
by Anonymous Monk on Aug 01, 2014 at 02:32 UTC
    shoot :) I had used the mutators to set seperator and those values and I couldn't get it to work :) thanks farang
[reply]
Re^6: Speeds vs functionality
by Jim (Curate) on Aug 01, 2014 at 18:23 UTC

    It works on a UTF-16 CSV file.

    use v5.14;
use strict;
use warnings;
use utf8;

use autodie qw( open close );
use Text::CSV_PP;

@ARGV == 1 or die "Usage: perl $0 <CSV file>\n";

my $file = shift;

open my $fh, '<:raw:perlio:encoding(UTF-16):crlf', $file;

my $csv = Text::CSV_PP->new({
    sep_char    => '🎬',
    quote_char  => '🎥',
    escape_char => '🎥',
    binary      => 1,
});

my @rows;

while (my $row = $csv->getline($fh)) {
    push @rows, $row;
}

$csv->eof() or $csv->error_diag();

close $fh;

binmode STDOUT, ':raw:perlio::encoding(UTF-16LE):crlf';

for my $row (@rows) {
    $row->[4] =~ s/\n\s*/, /g;

    printf "%-24s %s\n", $row->[0], $row->[4];
}

exit 0;

    See these nodes for an explanation of the UTF-16 PerlIO nonsense required on Microsoft Windows.

[reply]

In Section Meditations