Re^2: French Accents in perl

Dear monks, A million thanks for your help so far. I have taken the most practical advice given here and have added the two lines regarding encoding into the program. An illustrative program with the text I am trying to parse are included below. I am working in a windows environment, and use the dos prompt to call the script and .txt file that the above text is saved as. Unfortunately, the problem remains. Any further suggestions?

#!/usr/bin/perl
use open IN  => ':encoding(cp1252)';
use open OUT => ':utf8';
use Getopt::Std;
our ($opt_f);
getopts('f:');
if($opt_f){
    openFile ("$opt_f");   
    while (my $line = <FILE>) {    
        chomp ($line);

#the following line illustrates that what is being
#read in is transformed from the original text

                print $line;
                
#if the part in the reg exp below "(État)" is removed, then
#the reg exp matches and $1 is printed correctly, but not 
#with this kept in there
 
 if ($line=~ /Number\s+of\s+Observations\s+Used\s+(\d+)\s+(État)/){
        print "$1";
        print "$2";
}
}
}
[download]

Here is a small illustrative piece of text I am trying to parse using the -f option:

Le Système SAS                       243
                                    09:11 Monday, October 29, 2007

                       The PHREG Procedure

                   Informations sur le modèle

           Data Set                 WORK.PSEUDOCONTSAS
           Dependent Variable       time
           Censoring Variable       affstat
           Censoring Value(s)       0
           Ties Handling            DISCRETE


             Number of Observations Read        1162
             Number of Observations Used         332


                        État de convergence
[download]

Comment on Re^2: French Accents in perl Select or Download Code

Replies are listed 'Best First'.
Re^3: French Accents in perl by graff (Chancellor) on Nov 03, 2007 at 03:26 UTC
There are a few problems with that version of the script as posted: You should have just used a normal "open" statement instead of calling a subroutine that isn't defined. In my case at least (perl 5.8.6), the `use open OUT => ":utf8"` has no effect on the setting of STDOUT (which is already open when the script starts); I would use `binmode STDOUT, ":utf8"` You are reading the data file one line at a time, but your regex is trying to match a pattern that spans two lines, so the regex can never match successfully; you would either use two separate matches (one for each line), or else slurp the whole file into a single scalar in a single read operation (but that would change a lot of things). Here's a version that prints the matches, and the output is definitely valid utf8. If the results look wrong when you run it, that probably means you are using a display window that does not support utf8 characters. `#!/usr/bin/perl -w use strict; use Getopt::Std; our ($opt_f); getopts('f:'); if($opt_f){ open( FILE, "<:encoding(cp1252)", $opt_f); binmode STDOUT, ":utf8"; while (my $line = <FILE>) { chomp ($line); print "$line\n"; if ($line=~ /Number\s+of\s+Observations\s+Used\s+(\d+)/ or $line=~ /(État)/){ print "$1\n"; } } }` [download]	[reply] [d/l] [select]

Replies are listed 'Best First'.

Re^3: French Accents in perl
by graff (Chancellor) on Nov 03, 2007 at 03:26 UTC

You should have just used a normal "open" statement instead of calling a subroutine that isn't defined.
In my case at least (perl 5.8.6), the use open OUT => ":utf8" has no effect on the setting of STDOUT (which is already open when the script starts); I would use binmode STDOUT, ":utf8"
You are reading the data file one line at a time, but your regex is trying to match a pattern that spans two lines, so the regex can never match successfully; you would either use two separate matches (one for each line), or else slurp the whole file into a single scalar in a single read operation (but that would change a lot of things).

Here's a version that prints the matches, and the output is definitely valid utf8. If the results look wrong when you run it, that probably means you are using a display window that does not support utf8 characters.

#!/usr/bin/perl -w

use strict;
use Getopt::Std;

our ($opt_f);
getopts('f:');
if($opt_f){
    open( FILE, "<:encoding(cp1252)", $opt_f);
    binmode STDOUT, ":utf8";
    while (my $line = <FILE>) {
        chomp ($line);

        print "$line\n";

        if ($line=~ /Number\s+of\s+Observations\s+Used\s+(\d+)/ or
            $line=~ /(État)/){
            print "$1\n";
        }
    }
}
[download]

[reply]
[d/l]
[select]