Read perluniintro for more information on that matter.

If you have written a perl script yourself, please post it if it's fairly short. (If it's really long, try to show us a short script that demonstrates the same problem with your data.) If you are using a script that you downloaded from somewhere, can you tell us where you got that?

Do you know enough about character encodings to tell us what encoding is used by your input data? I would suggest trying to post a relevant sample of data with accents, but I'm not sure how reliable that would be (the data might get altered in a variety of ways that might be beyond your control) -- still, that might be worth a try.

Come to think of it, I wonder if the "bad" characters I see in your post are the ones you intended: I'm seeing U+00DE ("capital letter thorn") and U+2554 ("box drawings double down and right", which is typically 0xC9 in the old DOS code pages). You can look up unicode code points and get other useful info on encodings at http://www.unicode.org.

Anyway, it's hard for us to say what should be done until you give us more information (code and data).

See below for the code and data. Thanks.

use open IN  => ':encoding(cp1252)';
# Input files are in Windows CP1252.
# All input is converted to Perl's internal encoding
# (UTF-8) as it is read.

use open OUT => ':utf8';
# All output will be UTF-8.
[download]

Dear monks, A million thanks for your help so far. I have taken the most practical advice given here and have added the two lines regarding encoding into the program. An illustrative program with the text I am trying to parse are included below. I am working in a windows environment, and use the dos prompt to call the script and .txt file that the above text is saved as. Unfortunately, the problem remains. Any further suggestions?

#!/usr/bin/perl
use open IN  => ':encoding(cp1252)';
use open OUT => ':utf8';
use Getopt::Std;
our ($opt_f);
getopts('f:');
if($opt_f){
    openFile ("$opt_f");   
    while (my $line = <FILE>) {    
        chomp ($line);

#the following line illustrates that what is being
#read in is transformed from the original text

                print $line;
                
#if the part in the reg exp below "(État)" is removed, then
#the reg exp matches and $1 is printed correctly, but not 
#with this kept in there
 
 if ($line=~ /Number\s+of\s+Observations\s+Used\s+(\d+)\s+(État)/){
        print "$1";
        print "$2";
}
}
}
[download]

Here is a small illustrative piece of text I am trying to parse using the -f option:

Le Système SAS                       243
                                    09:11 Monday, October 29, 2007

                       The PHREG Procedure

                   Informations sur le modèle

           Data Set                 WORK.PSEUDOCONTSAS
           Dependent Variable       time
           Censoring Variable       affstat
           Censoring Value(s)       0
           Ties Handling            DISCRETE


             Number of Observations Read        1162
             Number of Observations Used         332


                        État de convergence
[download]

There are a few problems with that version of the script as posted:
• You should have just used a normal "open" statement instead of calling a subroutine that isn't defined.

• In my case at least (perl 5.8.6), the use open OUT => ":utf8" has no effect on the setting of STDOUT (which is already open when the script starts); I would use binmode STDOUT, ":utf8"

• You are reading the data file one line at a time, but your regex is trying to match a pattern that spans two lines, so the regex can never match successfully; you would either use two separate matches (one for each line), or else slurp the whole file into a single scalar in a single read operation (but that would change a lot of things).

Here's a version that prints the matches, and the output is definitely valid utf8. If the results look wrong when you run it, that probably means you are using a display window that does not support utf8 characters.

#!/usr/bin/perl -w

use strict;
use Getopt::Std;

our ($opt_f);
getopts('f:');
if($opt_f){
    open( FILE, "<:encoding(cp1252)", $opt_f);
    binmode STDOUT, ":utf8";
    while (my $line = <FILE>) {
        chomp ($line);

        print "$line\n";

        if ($line=~ /Number\s+of\s+Observations\s+Used\s+(\d+)/ or
            $line=~ /(État)/){
            print "$1\n";
        }
    }
}
[download]

...and they make whatever you say sound like "I surrender"? :-D
(oh, I'm gonna get downvotes for that one, but I couldn't resist)

--
Nothing to see here. Move along.

I'll probably get also downvoted, but I need to do it.

J'espère que vous vous rendez tous! Vive le français!

my reg1= /r\n/;#needed for UTF-8 cleaning
my $reg2 = /s/;#see http://unicode.org/reports/tr13/tr13-5.html

foreach (@words) {
    unless ($_ eq "s+"|$_ eq "0"|$_ eq $reg1|$_ eq $reg2){ #last clean
+ing
    ...
}
[download]

I'm sorry, but I must disagree. Based on what you've just said, the conclusion I would make is that you have not yet tried seriously to learn how perl 5.8 handles unicode; and you seem to be having trouble with regexes in general, as well.

A regex like /r\n/ (a letter "r" followed by a line-feed) has nothing whatsoever to do with "UTF-8 cleaning". I realize you probably meant /\r\n/ (CRLF), but that has nothing to do with utf8 cleaning either. (Oh, and there's a syntax error in that line.)

I don't know what you're getting at with the $reg2 = /s/ -- did you forget a backslash there as well? The foreach loop makes no sense at all, and I don't see how any of this (or Mark Davis's very useful essay on "Unicode Newline Guidelines") is at all relevant to accented letters.

The "\w" and "\W" work as expected on utf8 strings -- \w matches anything that is a letter or digit, no matter if it's ASCII, extended Latin, Greek, Arabic, Thai or whatever, and \W matches anything that is not a letter or digit. But you have to know for sure that you are applying the regex to a utf8 string that has been flagged internally by Perl as being a utf8 string. If you don't understand what that means, it's not surprising that you've had trouble with this stuff.