Regex Matching Unicode and Regex Classes

McA has asked for the wisdom of the Perl Monks concerning the following question:

Hi all, I have the following little script demonstrating a case (case 1 in the output below), which I can't explain to myself. So I hoped, someone can explain it to me or give the right hints.

#!/usr/bin/perl -CO
use strict;
use warnings;
use Encode;

use utf8;


my $a = 'ä';
print "UTF8-Flag: ", utf8::is_utf8($a) ? "Yes" : "No";
print "    matches word: ", $a =~ /\w/ ? "Yes\n" : "No\n";

my $b = encode("ISO-8859-1", $a);
print "UTF8-Flag: ", utf8::is_utf8($b) ? "Yes" : "No";
print "    matches word: ", $b =~ /\w/ ? "Yes\n" : "No\n";

use locale;

$a = 'ä';
print "UTF8-Flag: ", utf8::is_utf8($a) ? "Yes" : "No";
print "    matches word: ", $a =~ /\w/ ? "Yes\n" : "No\n";

$b = encode("ISO-8859-1", $a);
print "UTF8-Flag: ", utf8::is_utf8($b) ? "Yes" : "No";
print "    matches word: ", $b =~ /\w/ ? "Yes" : "No";
print "\n";
[download]

The output on a linux box with locale de_DE.UTF-8 and perl source code encoded in UTF-8 is:

UTF8-Flag: Yes    matches word: Yes
UTF8-Flag: No    matches word: No
UTF8-Flag: Yes    matches word: No
UTF8-Flag: No    matches word: No
[download]

It's the very first case I can't explain to me. Why is an unicode-flagged 'ä' matched against words when locale is not set explicitly?

Thanks in advance
Andreas

Comment on Regex Matching Unicode and Regex Classes Select or Download Code

Replies are listed 'Best First'.
Re: Regex Matching Unicode and Regex Classes by choroba (Cardinal) on Nov 02, 2011 at 14:05 UTC
Two days ago, I was having the same issues. My testing script is a bit more elaborate, but confirms your problem: `\w` does not work under `use locale`, whatever the locale is. If you need locale, use posix classes or unicode character properties (provided your locale is unicode based). Discussion on the details can be found in perlunicode and perllocale, but the relevant paragraphs are quite different in each version of Perl. #!/usr/bin/perl use warnings; use strict; my @p_locale = qw/C cs_CZ.UTF8 en_US.UTF8/; my @locale = ('no locale', 'use locale'); my @posix = (q(), 'use POSIX qw/locale_h/; setlocale LC_ALL,"C"', 'use POSIX qw/locale_h/; setlocale LC_ALL,"cs_CZ.UTF8"', ); for my $p_locale (@p_locale) { for my $locale (@locale) { for my $posix (@posix) { print "$p_locale $locale $posix\n"; open my $OUT, '>', 'l.perl' or die "$!"; print {$OUT} << " OUT"; $locale; $posix; my \@chars = qw/283 269 345 225 32 98 99 48 49 50 51 52 32 353 253 382 237/; my \$string = join q[], map chr, \@chars; binmode STDOUT, ':utf8'; my \@regex = (qr/(\\w+)/, qr/([[:alnum:]]+)/, qr/(\\p{W +ord}+)/); for my \$regex (\@regex) { print "\t\$1" while \$string =~ /\$regex/g; print "\n"; } print sort qw/ci ch/; print "\n"; OUT close $OUT; $ENV{LC_ALL} = $p_locale; system 'perl', 'l.perl'; } } } unlink 'l.perl'; [download]	[reply] [d/l] [select]
Re^2: Regex Matching Unicode and Regex Classes by McA (Priest) on Nov 02, 2011 at 14:58 UTC
Hi choroba, thank you for your answer. I just checked it on my platform. By the way: A program generating code, mutating and running it, is called a virus. ;-) Best regards Andreas	[reply]
Re^2: Regex Matching Unicode and Regex Classes by choroba (Cardinal) on Nov 04, 2011 at 10:34 UTC
Tested on several versions of Perl: In 5.10.1 and 5.12.1, \w does not work. In 5.14.1, \w and `[[:alnum:]]` do not work. Should I report a bug?	[reply] [d/l]
Re: Regex Matching Unicode and Regex Classes by moritz (Cardinal) on Nov 02, 2011 at 14:08 UTC
The default Unicode semantics just check the Unicode properties of a codepoint. "ä" is U+00E4 LATIN SMALL LETTER A WITH DIAERESIS and classified as a letter, so \w matches it. Perl 6 - second systems done right	[reply]
Re^2: Regex Matching Unicode and Regex Classes by McA (Priest) on Nov 02, 2011 at 14:27 UTC
Hi Moritz, but what is then the difference to the third case? Is the "default Unicode semantic" changed to something different when local is enabled? Why is "U+00E4 LATIN SMALL LETTER A WITH DIAERESIS" under locale something different than a letter which is part of a word? Best regards Andreas	[reply]
Re^3: Regex Matching Unicode and Regex Classes by moritz (Cardinal) on Nov 02, 2011 at 14:41 UTC
Short answer: because Unicode and locales don't mix. Long answer: Perl's support for locales comes from a time before the whole encoding/decoding business and Unicode support. So if locales are active, the locale-sensitive parts expect to act on bytes, not on decoded strings. Since the locale is not ISO-8859-1 but UTF-8, encoding to Latin-1 doesn't fix it for you. If anything, you'd need to encode to UTF-8 to see the \w matching ä, but even then I don't see it matching. So either my understanding of locales is very wrong, or perl is broken (or a mixture thereof). Perl 6 - second systems done right	[reply]
Re^4: Regex Matching Unicode and Regex Classes by McA (Priest) on Nov 02, 2011 at 14:48 UTC
Re^5: Regex Matching Unicode and Regex Classes by moritz (Cardinal) on Nov 02, 2011 at 14:53 UTC
Re^5: Regex Matching Unicode and Regex Classes by choroba (Cardinal) on Nov 02, 2011 at 14:58 UTC
Re^2: Regex Matching Unicode and Regex Classes by McA (Priest) on Nov 02, 2011 at 15:00 UTC
Moritz, thanky ou for your answers. Have a nice day. Best regards Andreas	[reply]


Perl: the Markov chain saw
	PerlMonks