french character in input file.

hotpower has asked for the wisdom of the Perl Monks concerning the following question:

Greetings Monks, I have a csv file which I'll use Perl to read in and do some manipulations and then output it in a new .csv file. Snippet of the input file: (Three lines each starting with the word "Map")

Map,Adèle,F,Blanc,BLANC Adèle,014596,014596,XEX-MRIJDVASSC-R3,,NOTE:BL
+ANC Adèle (014596)@EX%SMTP:adele.blanc@cc.com%X400:c=CA;a= ;p=OH;o=Mr
+ijdvasscAR3;s=BLANC;g=Adele;,p:046963@mail,Customer Srv Centre,WORKS/
+ SRV,SUPPORT/ SUPPORT,AXD3, + 1123,(603) 489-2365,(603) 254-7458,,,13
+ Wood Dr,Vancouver,ON,CAN,J7G 3Y4,/o=O:ou=AR3/337610,
Map,William,,Lemond,LEMOND Will,112477,112477,XEX-MC-A,,SMTP:Will.lemo
+nd@cc.com%LEMOND Will@EX%X400:c=CA;a= ;p=OH;o=MC;s=Lemond;g=Will;,,Ma
+int,,M STAT,AC,6224,,,,,,,,,,,
Map,Jacky,L,Mack,MACK Jacky,923636,923636,XEX-MNHYI-R3,,NOTES:MACK Jac
+ky (923636)@EX%SMTP:Jacky.Mack@cc.com%X400:c=CA;a= ;p=OH;o=MR3;s=MACK
+;g=Jacky;,p:@mail,Electrical,WORKS/ STS,OTT,K94, +,,,,,1501 Karten Ro
+ad,Ottawa,ON,CAN,N7Q B5L,/o=OH/u=MR3/k,
[download]

If I use the following code on the snippet of input file above, only the second and third line gets printed in the output file. The first line just got ignored. I am using Perl 5.8 on a Unix machine.

#!/usr/local/bin/perl
use strict;
use Text::CSV_XS;

my ($mday, $mon, $year) = (localtime(time))[3..5];
my $cdate = sprintf "%02d" x 3, $mon+1, $mday, $year%100;

my $infile = "inmail.csv"; 
my $outfile = "outmail${cdate}.csv"; 

if ( open( INFILE, "<:utf8", $infile ) ) {

    open (OUTFILE, ">:utf8", $outfile);
    
    my $csv = Text::CSV_XS->new();
    
    while (defined(my $line = <INFILE>)) {
        next if $. == 1; 
        chomp $line;
        if (my $status = $csv->parse($line)) {
            my @columns = $csv->fields;
            
            @columns = quotecolumn(@columns);
            
            for (9) {
                $columns[$_] =~ s/^.*SMTP://;
                $columns[$_] =~ s/\%.*$//;
            $columns[$_] =~ s/\".*$//; 
            }
            print OUTFILE join(",", @columns[1,2,3,4,5,9,11,12,13,15,1
+6,17,18,19,20,21,22,23,24]), "\n";
        } 
    }
    close (INFILE);
    close (OUTFILE);

   
}

sub quotecolumn {
    my @columns = @_;
    for (@columns) {
        if (/,/) {
            unless (/^"[^"]*"$/) {
                $_ = qq("$_");
            }
        }
    }
    return @columns;
}
[download]

If I remove è in Adèle in the first line, all three lines get printed in the output file. I am guess this is unicode problem? but I've tried using utf8, binmode but it doesn't work. I'm a beginner in Perl and any help would be appreciated. Thanks in advance!

Comment on french character in input file. Select or Download Code

Replies are listed 'Best First'.
Re: french character in input file. by jZed (Prior) on May 10, 2005 at 19:44 UTC
Text::CSV_XS treats accented characters as "binary" data. If you change your call to new() to be my $csv = Text::CSV_XS->new({binary=>1}), it should handle the file fine.	[reply]
Re^2: french character in input file. by hotpower (Initiate) on May 10, 2005 at 20:08 UTC
Thanks! It works great! Yes, sorry, I forgot to mention that the snippet of input file I posted was not at the beginning of the input file so the line that skips the first line has no effect this case.	[reply]
Re: french character in input file. by Roy Johnson (Monsignor) on May 10, 2005 at 19:40 UTC
Seems to me that the `next if $. == 1;` should be preventing processing of the first line, regardless of whether it has any freedom-hating* characters. * It's a joke, people. Caution: Contents may have been coded under pressure.	[reply] [d/l]