in reply to Parsing UTF-8 characters

The key is to ensure you're reading the input (and possibly also writing the output) using the right PerlIO layer — in this case, ':encoding(UTF-8)'. If you're using <> or <ARGV> (and possibly also STDOUT), then you may want to use the open pragma.

#!perl

use strict;
use warnings;
use open qw( :encoding(UTF-8) :std );

while (my $record = <>) {
    chomp $record;

    if ($record =~ m/^(pd\.\d+),([^,]+),([^,]+)$/) {
        my ($field1, $field2, $field3) = ($1, $2, $3);

        print "$field1\n";
        print "$field2\n";
        print "$field3\n";
    }
}

exit 0;

[download]

I recommend you use the Text::CSV module to parse the CSV records instead of a regular expression pattern. This will allow you to parse records that have literal commas in the text.

UPDATE: I changed ':utf8' to ':encoding(UTF-8)' based on something I read in Moritz's fine article titled Character Encodings in Perl.

Replies are listed 'Best First'.
Re^2: Parsing UTF-8 characters
by lordsll (Novice) on Oct 14, 2010 at 20:31 UTC
    Thank you, your example helped tremendously. The program now works exactly like I need. Thanks again.
[reply]

      Hopefully you took my advice to use Text::CSV instead of a regular expression pattern to parse the UTF-8 CSV records. The following script neatly parses the example record in your original post.

      #!perl

use strict;
use warnings;

use Text::CSV;

my $csv = Text::CSV->new({ auto_diag => 1, binary => 1 });

my $file = shift;

open my $fh, '<:encoding(UTF-8)', $file;

binmode STDOUT, ':encoding(UTF-8)';

my ($field1, $field2, $field3);

$csv->bind_columns(\($field1, $field2, $field3));

while ($csv->getline($fh)) {
    print "$field1\n";
    print "$field2\n";
    print "$field3\n";
}

exit 0;

      [download]
[reply]
[d/l]

In Section Seekers of Perl Wisdom