...regardless of UTF-8 flags

This note could be slightly misleading. Unicode properties matching only works with either Perl character/unicode strings (internally represented as UTF-8), in which case Perl's UTF-8 flag must be on, or with Latin1 as a special case, when the UTF-8 flag is off. In other words, the setting of the UTF-8 flag definitely is important. If it's off, the string is simply treated as Latin1, which might not always be appropriate.

Consider the unicode codepoint U+5F4C, for example (CJK Unified Ideograph). It's in the unicode properties category "OtherLetter", and thus also in "Alphabetic", which Perl does identify correctly when it's represented as a Perl character string with the UTF-8 flag on. When the same character is represented in some other (non-Latin1) encoding, e.g. Shift-JIS, Perl has no way of telling that it's in fact a letter. It just treats the respective bytes (i.e. the Shift-JIS two-byte representation 0x9c 0x5c) as Latin1, looks up those codepoints in the unicode database (which gives "STRING TERMINATOR" (category "Control") for U+009C, and "BACKSLASH" (category "OtherPunctuation") for U+005C), and reaches the wrong conclusion, as of course none of those qualify as "Alphabetic"... 

use Encode;
use Devel::Peek;

print "U+5F4C as a regular Perl character string (UTF-8 flag on):\n";
my $str = "\x{5F4C}";  
Dump $str;
printf "=> is%s alphabetic\n\n", $str =~ /\p{IsAlpha}/ ? "":"n't";

print "same character in Shift-JIS legacy encoding (UTF-8 flag off):\n
+";
$str = encode("shiftjis", $str);
Dump $str;
printf "=> is%s alphabetic\n\n", $str =~ /\p{IsAlpha}/ ? "":"n't";

[download]
U+5F4C as a regular Perl character string (UTF-8 flag on):
SV = PV(0x629098) at 0x645530
  REFCNT = 1
  FLAGS = (PADMY,POK,pPOK,UTF8)
  PV = 0x682d40 "\345\275\214"\0 [UTF8 "\x{5f4c}"]
  CUR = 3
  LEN = 8
=> is alphabetic

same character in Shift-JIS legacy encoding (UTF-8 flag off):
SV = PV(0x629098) at 0x645530
  REFCNT = 1
  FLAGS = (PADMY,POK,pPOK)
  PV = 0x75b5d0 "\234\\"\0
  CUR = 2
  LEN = 8
=> isn't alphabetic

[download]

In reply to Re^2: Why [[:alpha:]] doesn't involve diacritic characters in replace expression? by almut
in thread Why [[:alpha:]] doesn't involve diacritic characters in replace expression? by wk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.