in reply to French Accents in perl

You want to start your script with something like this (assuming that your input file is encoded as Windows' Code Page 1252): 
use open IN  => ':encoding(cp1252)';
# Input files are in Windows CP1252.
# All input is converted to Perl's internal encoding
# (UTF-8) as it is read.

use open OUT => ':utf8';
# All output will be UTF-8.

[download]
Substitute cp1252 and utf8 for your input encoding and desired output encoding respectively.

Replies are listed 'Best First'.
Re^2: French Accents in perl
by audoushka (Novice) on Nov 01, 2007 at 20:11 UTC
    Dear monks, A million thanks for your help so far. I have taken the most practical advice given here and have added the two lines regarding encoding into the program. An illustrative program with the text I am trying to parse are included below. I am working in a windows environment, and use the dos prompt to call the script and .txt file that the above text is saved as. Unfortunately, the problem remains. Any further suggestions? 
    #!/usr/bin/perl
use open IN  => ':encoding(cp1252)';
use open OUT => ':utf8';
use Getopt::Std;
our ($opt_f);
getopts('f:');
if($opt_f){
    openFile ("$opt_f");   
    while (my $line = <FILE>) {    
        chomp ($line);

#the following line illustrates that what is being
#read in is transformed from the original text

                print $line;
                
#if the part in the reg exp below "(État)" is removed, then
#the reg exp matches and $1 is printed correctly, but not 
#with this kept in there
 
 if ($line=~ /Number\s+of\s+Observations\s+Used\s+(\d+)\s+(État)/){
        print "$1";
        print "$2";
}
}
}

    [download]
    Here is a small illustrative piece of text I am trying to parse using the -f option: 
    Le Système SAS                       243
                                    09:11 Monday, October 29, 2007

                       The PHREG Procedure

                   Informations sur le modèle

           Data Set                 WORK.PSEUDOCONTSAS
           Dependent Variable       time
           Censoring Variable       affstat
           Censoring Value(s)       0
           Ties Handling            DISCRETE


             Number of Observations Read        1162
             Number of Observations Used         332


                        État de convergence

    [download]
[reply]
[d/l]
[select]
      There are a few problems with that version of the script as posted:

      • You should have just used a normal "open" statement instead of calling a subroutine that isn't defined.

      • In my case at least (perl 5.8.6), the use open OUT => ":utf8" has no effect on the setting of STDOUT (which is already open when the script starts); I would use binmode STDOUT, ":utf8"

      • You are reading the data file one line at a time, but your regex is trying to match a pattern that spans two lines, so the regex can never match successfully; you would either use two separate matches (one for each line), or else slurp the whole file into a single scalar in a single read operation (but that would change a lot of things).

      Here's a version that prints the matches, and the output is definitely valid utf8. If the results look wrong when you run it, that probably means you are using a display window that does not support utf8 characters. 

      #!/usr/bin/perl -w

use strict;
use Getopt::Std;

our ($opt_f);
getopts('f:');
if($opt_f){
    open( FILE, "<:encoding(cp1252)", $opt_f);
    binmode STDOUT, ":utf8";
    while (my $line = <FILE>) {
        chomp ($line);

        print "$line\n";

        if ($line=~ /Number\s+of\s+Observations\s+Used\s+(\d+)/ or
            $line=~ /(État)/){
            print "$1\n";
        }
    }
}

      [download]
[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom