perl unicode docs

7stud has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks,

A lot of unicode beginners believe that Unicode characters and UTF-8 characters are the same thing. If they delve into the subject a little further, it becomes patently obvious that a Unicode character and a UTF-8 character are not the same thing. In fact, you soon learn that you have to encode a Unicode character with a "UTF-8 translator" to produce a UTF-8 character.

Some time after acquiring such powerful knowledge, a beginner might even attempt to read the unicode docs, like perlunicode, to get clarification on an issue, but then come across statements like this:

Regular Expressions
           The regular expression compiler produces polymorphic opcodes.  That
           is, the pattern adapts to the data and automatically switches to
           the Unicode character scheme when presented with data that is
           internally encoded in UTF-8 -- or instead uses a traditional byte
           scheme when presented with byte data.

The fact of the matter is, if the pattern switched to the Unicode character scheme then the pattern couldn't possibly match a single character in a UTF-8 string.

I think it would be nice if the person who wrote the perlunicode docs had adhered to the basic tenants of unicode when describing perl's state of unicode awareness. Or, was the idea to dumb down the docs and present factually incorrect descriptions so that beginners who think that Unicode characters are the same as UTF-8 characters are not confused?

Comment on perl unicode docs

Replies are listed 'Best First'.
Re: perl unicode docs by moritz (Cardinal) on Mar 27, 2010 at 12:16 UTC
The fact of the matter is, if the pattern switched to the Unicode character scheme then the pattern couldn't possibly match a single character in a UTF-8 string. Why not? If by "a single character" you mean a codepoint encoded it multiple bytes, then yes, that's the default mode. If you by "a single character" you mean a byte of a multi-byte UTF-8 sequence, then yes, even that's possible (with the `\C` escape. Yes, it's... weird, but it is implemented). (But nowhere in the unicode docs I can find an indication that this is meant). I think it would be nice if the person who wrote the perlunicode docs had adhered to the basic tenants of unicode when describing perl's state of unicode awareness. And where did he not? By the way, if you find places where the docs need improvement, don't whine about it, but submit patches. Or, was the idea to dumb down the docs and present factually incorrect descriptions so that beginners who think that Unicode characters are the same as UTF-8 characters are not confused? I don't think so. I also don't see how anything of the docs is factually incorrect. Perl 6 - links to (nearly) everything that is Perl 6.	[reply] [d/l]
Re^2: perl unicode docs by 7stud (Deacon) on Mar 27, 2010 at 13:09 UTC
The fact of the matter is, if the pattern switched to the Unicode character scheme then the pattern couldn't possibly match a single character in a UTF-8 string. Why not? Because a Unicode character and a UTF-8 character are different entities. If the regex is looking for a Unicode character, the regex will never find one in a string that contains only UTF-8 characters. By the way, if you find places where the docs need improvement, don't whine about it, but submit patches. Wouldn't it be best to discuss proposed changes before submitting anything? In any case, I spent 7 yours one day trying to do just that--without success, and I will never, ever try to interface with that archaic system ever again. I don't think so. I also don't see how anything of the docs is factually incorrect. Well, you are smarter than me, so I guess that's the end of that.	[reply]
Re^3: perl unicode docs by moritz (Cardinal) on Mar 27, 2010 at 14:55 UTC
Because a Unicode character and a UTF-8 character are different entities. How so? What do you mean by "UTF-8 character"? I read that as "one Unicode character encoded in UTF-8", not as "one byte". Wouldn't it be best to discuss proposed changes before submitting anything? Yes. But that's not what your question sounded like. It sounded more like "is it just me, or were the original authors jerks?" I spent 7 yours one day trying to do just that--without success, and I will never, ever try to interface with that archaic system ever again. Now it would be interesting to here what you man by "that archaic system". The source control system is quite modern. The p5p list is another matter, but so far I never had problems getting doc patches applied. Perl 6 - links to (nearly) everything that is Perl 6.	[reply]
Re^2: perl unicode docs by 7stud (Deacon) on Mar 27, 2010 at 13:09 UTC
double post	[reply]
Re: perl unicode docs by ikegami (Patriarch) on Mar 27, 2010 at 15:37 UTC
A lot of unicode beginners believe that Unicode characters and UTF-8 characters are the same thing. Unicode is a character set. It has thousands of characters, far more to allow every character to be stored in a byte. UTF-8 is a way of storing Unicode characters, since the system deals with bytes. Some important background: Perl's documentation regarding unicode has historically been poor, stemming from Perl 5.6's failed attempt at unicode support. That paragraph in question attempts to describe a bug in Perl. Confusion is natural. The usual Perl lingo for "Unicode character scheme" is "Unicode semantics". It refers to the state when `/\w/` matches "é" and other iso-8859-1 adorned letters and when `\s` matches NBSP. The regex engine behaves that way in response to an internal state, thus the bug. `uc` and similar are also affected. Unforunately, we're stuck stuck with the bug. People expect \w to only match ASCII letters and people expect to match any Unicode letters, and it usually works for both sets of people. Fixing the bug would mean it would always do one or the other. The bug was therefore fixed via a pragma. If your program has `use 5.012;` or `use feature 'unicode_strings';`, unicode semantics will always be on, and the paragraph becomes The regular expression compiler produces polymorphic opcodes. That is, the pattern adapts to the data and works regardless of the internal encoding of the data.	[reply] [d/l] [select]
Re: perl unicode docs by grantm (Parson) on Mar 28, 2010 at 23:27 UTC
The important point is that a Unicode code point is an abstract concept. It's just a mapping between a character and a number. A concrete instantiation of a Unicode code point requires that the number be expressed in some encoding. Perl's internal representation of character strings (as opposed to byte strings) uses UTF-8 (with some minor differences which I will ignore here). Therefore if you want to match a regular expression against a character string, both the string and the regular expression must be expressed in a way that allows Perl to represent them in UTF8. When this is done, Perl will be able to recognise characters even when they are more than one byte long. An example may help. Consider this variable assignment: `my $currency = "\x{20AC}"; # The Euro symbol` [download] The `$currency` variable now contains a UTF8 representation of unicode character U+20AC. Although `length($currency)` will return 1 (since the string is 1 character long), the string will actually contain the three bytes: E2 82 AC So this will match: `perl -le '"\x{20AC}" =~ /\x{20AC}/ && print "Match!"'` and this will not `perl -le '"\xE2\x82\xAC" =~ /\x{20AC}/ && print "Match!"'` even though the two strings have identical contents, in one case it's a character string and in the other case it's a byte string.	[reply] [d/l] [select]
A reply falls below the community's threshold of quality. You may see it by logging in.