Re: Why [[:alpha:]] doesn't involve diacritic characters in replace expression?

It doesn't explain why you are getting what you get, but the use of \w, \W, [:alpha:], [:word:], etc is problematic in Perl, if the string you're matching against contains characters in the range 128-255, and no characters above. Then \w, [:alpha:] and such may match the accented characters, depending whether the string matched against has the UTF-8 flag (for that string) on or not. That is, it will match if the UTF-8 flag is on. If it's off, than in some cases if the pattern matched with has the UTF-8 flag on \w will match against accented characters - but not always. And, finally, if there's no UTF-8 it will be your locale that decides it - if you have an active locale.

Much safer is to use the appropriate Unicode attributes. \p{IsAlpha} will match all characters marked 'alpha' in the Unicode database, regardless of UTF-8 flags or locales.

my $_ = "See on üks täppidega lause!";
for (split //) {print if /\p{IsAlpha}/}
say "";
$_ =~ s/\P{IsAlpha}//g;
say;
__END__
Seeonükstäppidegalause
Seeonükstäppidegalause
[download]

Comment on Re: Why [[:alpha:]] doesn't involve diacritic characters in replace expression? Download Code

Replies are listed 'Best First'.
Re^2: Why [[:alpha:]] doesn't involve diacritic characters in replace expression? by moritz (Cardinal) on Oct 22, 2008 at 15:09 UTC
It doesn't explain why you are getting what you get, but the use of \w, \W, `[:alpha:]`, `[:word:]`, etc is problematic in Perl, if the string you're matching against contains characters in the range 128-255, and no characters above To guard against that, you can use Unicode::Semantics, or utf8::upgrade on the string prior to matching. That said I'd stay with the Unicode properties in regexes, and only revert to locales if you really need language dependent behavior. For a task like identifying printable and non-printable characters, Unicode is most likely the better choice.	[reply] [d/l] [select]
Re^2: Why [[:alpha:]] doesn't involve diacritic characters in replace expression? by almut (Canon) on Oct 22, 2008 at 18:59 UTC
...regardless of UTF-8 flags This note could be slightly misleading. Unicode properties matching only works with either Perl character/unicode strings (internally represented as UTF-8), in which case Perl's UTF-8 flag must be on, or with Latin1 as a special case, when the UTF-8 flag is off. In other words, the setting of the UTF-8 flag definitely is important. If it's off, the string is simply treated as Latin1, which might not always be appropriate. Consider the unicode codepoint U+5F4C, for example (CJK Unified Ideograph). It's in the unicode properties category "OtherLetter", and thus also in "Alphabetic", which Perl does identify correctly when it's represented as a Perl character string with the UTF-8 flag on. When the same character is represented in some other (non-Latin1) encoding, e.g. Shift-JIS, Perl has no way of telling that it's in fact a letter. It just treats the respective bytes (i.e. the Shift-JIS two-byte representation `0x9c 0x5c`) as Latin1, looks up those codepoints in the unicode database (which gives "STRING TERMINATOR" (category "Control") for U+009C, and "BACKSLASH" (category "OtherPunctuation") for U+005C), and reaches the wrong conclusion, as of course none of those qualify as "Alphabetic"... `use Encode; use Devel::Peek; print "U+5F4C as a regular Perl character string (UTF-8 flag on):\n"; my $str = "\x{5F4C}"; Dump $str; printf "=> is%s alphabetic\n\n", $str =~ /\p{IsAlpha}/ ? "":"n't"; print "same character in Shift-JIS legacy encoding (UTF-8 flag off):\n +"; $str = encode("shiftjis", $str); Dump $str; printf "=> is%s alphabetic\n\n", $str =~ /\p{IsAlpha}/ ? "":"n't";` [download] `U+5F4C as a regular Perl character string (UTF-8 flag on): SV = PV(0x629098) at 0x645530 REFCNT = 1 FLAGS = (PADMY,POK,pPOK,UTF8) PV = 0x682d40 "\345\275\214"\0 [UTF8 "\x{5f4c}"] CUR = 3 LEN = 8 => is alphabetic same character in Shift-JIS legacy encoding (UTF-8 flag off): SV = PV(0x629098) at 0x645530 REFCNT = 1 FLAGS = (PADMY,POK,pPOK) PV = 0x75b5d0 "\234\\"\0 CUR = 2 LEN = 8 => isn't alphabetic` [download]	[reply] [d/l] [select]
Re^2: Why [[:alpha:]] doesn't involve diacritic characters in replace expression? by wk (Novice) on Oct 22, 2008 at 16:02 UTC
Thank you, JavaFan! That's it. I looked for those Unicode markups and thought: no way this could help me, it's the same story... :)) And yes, there is problem with diacritics, which are in Latin1-charset but not those which are above it. For example 'š' worked great. Thank you for the solution! Kõike hääd. WK	[reply]