lordsll has asked for the wisdom of the Perl Monks concerning the following question:

I need to read and parse a file having at least one field of utf8 characters. The fields are separated with commas, but the regex does not seem to recognize the comma as separator. I do have the 'use utf8'. The utf8 comes in as actual character. The input is of the format: pd.1030505718, Прасковья Сергеева,400005W12ZMtLW4bVG1u8000 400005W12ZMtLW4bVG1vg000 400005W12ZMtLW4bVG1v8000 400003Th2ZItG14H3aKWJ000 and I am trying to parse using the following regex: if (/(pd.\d+).(\X+ ).(\w+|\d+| )/) { I am fairly new to perl and would appreciate any help, Thanks for your help,

Replies are listed 'Best First'.
Re: Parsing UTF-8 characters
by Jim (Curate) on Oct 14, 2010 at 17:57 UTC

    The key is to ensure you're reading the input (and possibly also writing the output) using the right PerlIO layer — in this case, ':encoding(UTF-8)'. If you're using <> or <ARGV> (and possibly also STDOUT), then you may want to use the open pragma.

    #!perl use strict; use warnings; use open qw( :encoding(UTF-8) :std ); while (my $record = <>) { chomp $record; if ($record =~ m/^(pd\.\d+),([^,]+),([^,]+)$/) { my ($field1, $field2, $field3) = ($1, $2, $3); print "$field1\n"; print "$field2\n"; print "$field3\n"; } } exit 0;

    I recommend you use the Text::CSV module to parse the CSV records instead of a regular expression pattern. This will allow you to parse records that have literal commas in the text.

    UPDATE: I changed ':utf8' to ':encoding(UTF-8)' based on something I read in Moritz's fine article titled Character Encodings in Perl.

      Thank you, your example helped tremendously. The program now works exactly like I need. Thanks again.

        Hopefully you took my advice to use Text::CSV instead of a regular expression pattern to parse the UTF-8 CSV records. The following script neatly parses the example record in your original post.

        #!perl use strict; use warnings; use Text::CSV; my $csv = Text::CSV->new({ auto_diag => 1, binary => 1 }); my $file = shift; open my $fh, '<:encoding(UTF-8)', $file; binmode STDOUT, ':encoding(UTF-8)'; my ($field1, $field2, $field3); $csv->bind_columns(\($field1, $field2, $field3)); while ($csv->getline($fh)) { print "$field1\n"; print "$field2\n"; print "$field3\n"; } exit 0;
Re: Parsing UTF-8 characters
by moritz (Cardinal) on Oct 14, 2010 at 17:48 UTC
    Are you decoding your incoming data with Encode::decode_utf8? Or do you set up an ':encoding(UTF-8)' IO layer?

    See also: Encodings and Unicode in Perl.

    Perl 6 - links to (nearly) everything that is Perl 6.