Re^3: use locale behavior depends on charset of locale?

The behaviour of \w shouldn't change based on whether it's in a char class or not.

But running with -Dr shows that these regexes compiled differently. And it looks like the problem is not in IO, I've tried the following:

#use locale;
use utf8;
$a = 'é';
$a =~ m/\w/ and print "first re match\n"; 
$a =~ m/[\w]/ and print "second re match\n";
[download]

if use locale is commented out it outputs:

first re match
second re match
[download]

if use locale uncommented only:

second re match
[download]

My locale en_US.UTF-8 BTW.

debugperl -Dr shows the following:

Without locale:
Compiling REx "\w"
Final program:
   1: ALNUM (2)
   2: END (0)
stclass ALNUM minlen 1 
Compiling REx "[\w]"
Final program:
   1: ANYOF[0-9A-Z_a-z+utf8::IsWord] (13)
  13: END (0)
stclass ANYOF[0-9A-Z_a-z+utf8::IsWord] minlen 1 

with locale:
Compiling REx "\w"
Final program:
   1: ALNUML (2)
   2: END (0)
stclass ALNUML minlen 1 
Compiling REx "[\w]"
Final program:
   1: ANYOF{loc}[\w+utf8::IsWord] (13)
  13: END (0)
stclass ANYOF{loc}[\w+utf8::IsWord] minlen 1
[download]

I don't really understand this output very well, but it looks like it treats this regexes differently.

Comment on Re^3: use locale behavior depends on charset of locale? Select or Download Code

Replies are listed 'Best First'.
Re^4: use locale behavior depends on charset of locale? by ikegami (Patriarch) on Jul 10, 2009 at 18:35 UTC
And it looks like the problem is not in IO Correct. That's what my changes show. I don't really understand this output very well, but it looks like it treats this regexes differently. It's not a problem that `/\w/` and `/[\w]/` compile differently. It's a problem that they don't compile to something equivalent. `use locale; utf8::upgrade( my $s = chr(0xC9) ); # e-acute print "Outside char class: ", $s =~ m/\w/ ? "" : "no ", "match\n"; print "Inside char class: ", $s =~ m/[\w]/ ? "" : "no ", "match\n";` [download] `LANG=en_CA.utf8 perl test.pl Outside char class: no match Inside char class: match` [download]	[reply] [d/l] [select]

Replies are listed 'Best First'.

Re^4: use locale behavior depends on charset of locale?
by ikegami (Patriarch) on Jul 10, 2009 at 18:35 UTC

And it looks like the problem is not in IO

Correct. That's what my changes show.

I don't really understand this output very well, but it looks like it treats this regexes differently.

It's not a problem that /\w/ and /[\w]/ compile differently. It's a problem that they don't compile to something equivalent.

use locale;
utf8::upgrade( my $s = chr(0xC9) );   # e-acute
print "Outside char class: ", $s =~ m/\w/   ? "" : "no ", "match\n";
print "Inside char class:  ", $s =~ m/[\w]/ ? "" : "no ", "match\n";
[download]

LANG=en_CA.utf8 perl test.pl
Outside char class: no match
Inside char class:  match
[download]

[reply]
[d/l]
[select]