in reply to UTF8 versus \w in pattern matching

Problem #1: You didn't tell Perl the source file is encoded using use utf8;.

Problem #2: You didn't tell Perl how to encode the output for your terminal using something like use open ':std', ':encoding(UTF-8)';.

Finally, you mention \w. Because of a bug, \w doesn't always match characters in the U+7F..U+FF range. This bug is fixed with use 5.014;. That said, you actually used \p{Word}, which isn't affected by this bug.

Seeking work! You can reach me at ikegami@adaelis.com

Replies are listed 'Best First'.
Re^2: UTF8 versus \w in pattern matching
by mldvx4 (Hermit) on Jul 07, 2021 at 03:45 UTC

    Thanks. That part about the U+7F..U+FF range explains things. I see that m/^([\/\-\_\.\p{Word}\x7f-\xff]+)$/ matches, and m/^([\/\-\_\.\p{Word}]+)$/ does not. I presume that is because the data upstream might really be ISO-8859-15 and not UTF-8? Should I try to convert the U+7F..U+FF range into UTF-8 before further processing? If so, how?

[reply]
[d/l]
[select]

      Read again. The part about U+7F..U+FF applies to \w, but not to \p{Word}. 

      # Sometimes \w matches U+E9.
$ perl -Mfeature=say -e'say "\xE9" =~ /^\w/ || 0'
0

# Sometimes it doesn't.
$ perl -Mfeature=say -e'say "\xE9\x{2660}" =~ /^\w/ || 0'
1

# \w always matches U+E9 with "use 5.014;".
$ perl -Mfeature=say -e'use 5.014; say "\xE9" =~ /^\w/ || 0'
1

# \p{Word} always matches U+E9, period.
$ perl -Mfeature=say -e'say "\xE9" =~ /^\p{Word}/ || 0'
1

      [download]

      Your questions make absolutely no sense since you're not using \w. And I said the fix for that bug was to add use 5.014;, not to convert the input.

      Seeking work! You can reach me at ikegami@adaelis.com

[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom