in reply to Re: Why [[:alpha:]] doesn't involve diacritic characters in replace expression?
in thread Why [[:alpha:]] doesn't involve diacritic characters in replace expression?
...regardless of UTF-8 flags
This note could be slightly misleading. Unicode properties matching only works with either Perl character/unicode strings (internally represented as UTF-8), in which case Perl's UTF-8 flag must be on, or with Latin1 as a special case, when the UTF-8 flag is off. In other words, the setting of the UTF-8 flag definitely is important. If it's off, the string is simply treated as Latin1, which might not always be appropriate.
Consider the unicode codepoint U+5F4C, for example (CJK Unified Ideograph). It's in the unicode properties category "OtherLetter", and thus also in "Alphabetic", which Perl does identify correctly when it's represented as a Perl character string with the UTF-8 flag on. When the same character is represented in some other (non-Latin1) encoding, e.g. Shift-JIS, Perl has no way of telling that it's in fact a letter. It just treats the respective bytes (i.e. the Shift-JIS two-byte representation 0x9c 0x5c) as Latin1, looks up those codepoints in the unicode database (which gives "STRING TERMINATOR" (category "Control") for U+009C, and "BACKSLASH" (category "OtherPunctuation") for U+005C), and reaches the wrong conclusion, as of course none of those qualify as "Alphabetic"...
use Encode; use Devel::Peek; print "U+5F4C as a regular Perl character string (UTF-8 flag on):\n"; my $str = "\x{5F4C}"; Dump $str; printf "=> is%s alphabetic\n\n", $str =~ /\p{IsAlpha}/ ? "":"n't"; print "same character in Shift-JIS legacy encoding (UTF-8 flag off):\n +"; $str = encode("shiftjis", $str); Dump $str; printf "=> is%s alphabetic\n\n", $str =~ /\p{IsAlpha}/ ? "":"n't";
U+5F4C as a regular Perl character string (UTF-8 flag on): SV = PV(0x629098) at 0x645530 REFCNT = 1 FLAGS = (PADMY,POK,pPOK,UTF8) PV = 0x682d40 "\345\275\214"\0 [UTF8 "\x{5f4c}"] CUR = 3 LEN = 8 => is alphabetic same character in Shift-JIS legacy encoding (UTF-8 flag off): SV = PV(0x629098) at 0x645530 REFCNT = 1 FLAGS = (PADMY,POK,pPOK) PV = 0x75b5d0 "\234\\"\0 CUR = 2 LEN = 8 => isn't alphabetic
|
|---|