Re: detecting non-ascii chars in a string

I tried with Chinese characters, each has three bytes when encoded in utf8, the same as Japanese. Couple of things from my testing (on Win32):

About ord: ord simply takes whatever you give it as a byte, and returns the value of that byte. This has nothing to do with the language or the encoding schema. It is byte semantics, doesn't matter whether you specify use utf8 or not. This makes sense.
About encoding: The encding schema determines the value returns from ord. For example, if you use utf8, then each Japanese character is encoded as a stream of three bytes, following this value pattern: 1110nnnn, 10nnnnnn, 10nnnnnn.
About length: Tested in Perl 5.8.0. My data file is Chinese encoded in utf8. It doesn't matter whether I specify 'use utf8' in the script or not, the result returned is always the same, which is the number of bytes, so this is also byte semantics, again it makes sense.
About split: with Perl 5.6.1, if I specify use utf8, then the character semantics is forced. For example, split(//, $str) returns a list of strings, and each of the strings is three bytes in length, which is the length of one Chinese character. If I don't specify use utf8, it follows byte semantics.

It does make sense, however is broken in Perl 5.8.0. A feature? no way, I believe this is a bug in Perl 5.8.0.

Comment on Re: detecting non-ascii chars in a string

Replies are listed 'Best First'.
Re: Re: detecting non-ascii chars in a string by grantm (Parson) on Dec 05, 2002 at 08:51 UTC
I think you have something wrong with your test data. The ord() function most definitely does correctly handle multi-byte UTF-8 characters. For example this code: `use utf8; my $euro = "\x{20AC}"; print ord($euro); print length($euro);` [download] correctly prints 8364 and 1 (even under 5.6). In UTF-8, the Euro symbol encodes to the three bytes 0xE2 0x82 0xAC but this code: `use utf8; my $binary = "\xE2\x82\xAC"; print ord($binary); print length($binary);` [download] prints out 226 and 3. Both strings consist of the same three bytes, but $euro is flagged internally as a UTF-8 string whereas $binary is not. Remember, Perl has to deal with both character data and binary data. Just because Perl sees the 0xE2 0x82 0xAC sequence of bytes does not mean it should always treat it as a single character. Here are a few ways to enter UTF-8 encoded non-ASCII characters: Use a UTF-8 aware editor (eg: VIM) and just type them in to string literals (or even variable names) Use the Unicode escape sequence \x{XXXX} in a double quoted string (see above) Read some data in from an XML file using a proper XML Parser module For more details see the utf8 man page.	[reply] [d/l] [select]

Replies are listed 'Best First'.

Re: Re: detecting non-ascii chars in a string
by grantm (Parson) on Dec 05, 2002 at 08:51 UTC

I think you have something wrong with your test data. The ord() function most definitely does correctly handle multi-byte UTF-8 characters. For example this code:

use utf8;
my $euro = "\x{20AC}";
print ord($euro);
print length($euro);
[download]

correctly prints 8364 and 1 (even under 5.6). In UTF-8, the Euro symbol encodes to the three bytes 0xE2 0x82 0xAC but this code:

use utf8;
my $binary = "\xE2\x82\xAC";
print ord($binary);
print length($binary);
[download]

prints out 226 and 3. Both strings consist of the same three bytes, but $euro is flagged internally as a UTF-8 string whereas $binary is not.

Remember, Perl has to deal with both character data and binary data. Just because Perl sees the 0xE2 0x82 0xAC sequence of bytes does not mean it should always treat it as a single character.

Here are a few ways to enter UTF-8 encoded non-ASCII characters:

Use a UTF-8 aware editor (eg: VIM) and just type them in to string literals (or even variable names)
Use the Unicode escape sequence \x{XXXX} in a double quoted string (see above)
Read some data in from an XML file using a proper XML Parser module

For more details see the utf8 man page.

[reply]
[d/l]
[select]