blahblahblah has asked for the wisdom of the Perl Monks concerning the following question:

Hi,
I need a way to tell if a given string contains non-ascii characters (specifically, Japanese double-byte characters). I thought I might be able to split the string and then call ord on each character, and tell from the results of ord whether it's ascii or not, like this: 
@chars = split '', $string;
for my $char (@chars)
{
   my $ord = ord $char;
   print "$char   $ord\n";
}

[download]
But so far either the split seems to be giving me the wrong number of characters, or ord is trying to force them into an incorrect ascii value.
Anyone have a good way to do this? The next thing I'm going to do is peek inside the Jcode module to see if it has a good trick I can use -- I'll post whatever I come up with.

Replies are listed 'Best First'.
Re: detecting non-ascii chars in a string
by Zaxo (Archbishop) on Dec 05, 2002 at 02:30 UTC

    The answer depends on your version of perl. This becomes much more powerful, if more difficult, witn the unicode support available in recent Perl.

    For perl without unicode, you only need to define a character class for the 8-bit numbers of interest: my $NotASCII = '[\x80-\xff]'; catches them all. Don't put that between \Q\E in a regex, you want it to be interpolated with the metacharacters unescaped. It is easy to get exactly what you want with a unicode perl: 

    use utf8;
print 'Hiragana or Katakana detected!', $/
    if /[\p{InHiragana}\p{InKatakana}]/;

    [download]
    You will get odd results from one perl to the next by counting characters. For example, the length builtin uses character semantics, so the results differ depending on whether utf8 is in force or not. The use byte(); pragmatic will allow you to count bytes with byte::length.

    After Compline,
    Zaxo

[reply]
[d/l]
[select]
Re: detecting non-ascii chars in a string
by pg (Canon) on Dec 05, 2002 at 02:16 UTC
    I tried with Chinese characters, each has three bytes when encoded in utf8, the same as Japanese. Couple of things from my testing (on Win32):

    • About ord: ord simply takes whatever you give it as a byte, and returns the value of that byte. This has nothing to do with the language or the encoding schema. It is byte semantics, doesn't matter whether you specify use utf8 or not. This makes sense.
    • About encoding: The encding schema determines the value returns from ord. For example, if you use utf8, then each Japanese character is encoded as a stream of three bytes, following this value pattern: 1110nnnn, 10nnnnnn, 10nnnnnn.
    • About length: Tested in Perl 5.8.0. My data file is Chinese encoded in utf8. It doesn't matter whether I specify 'use utf8' in the script or not, the result returned is always the same, which is the number of bytes, so this is also byte semantics, again it makes sense.
    • About split: with Perl 5.6.1, if I specify use utf8, then the character semantics is forced. For example, split(//, $str) returns a list of strings, and each of the strings is three bytes in length, which is the length of one Chinese character. If I don't specify use utf8, it follows byte semantics.

      It does make sense, however is broken in Perl 5.8.0. A feature? no way, I believe this is a bug in Perl 5.8.0.
[reply]

      I think you have something wrong with your test data. The ord() function most definitely does correctly handle multi-byte UTF-8 characters. For example this code:

      use utf8;
my $euro = "\x{20AC}";
print ord($euro);
print length($euro);

      [download]

      correctly prints 8364 and 1 (even under 5.6). In UTF-8, the Euro symbol encodes to the three bytes 0xE2 0x82 0xAC but this code:

      use utf8;
my $binary = "\xE2\x82\xAC";
print ord($binary);
print length($binary);

      [download]

      prints out 226 and 3. Both strings consist of the same three bytes, but $euro is flagged internally as a UTF-8 string whereas $binary is not.

      Remember, Perl has to deal with both character data and binary data. Just because Perl sees the 0xE2 0x82 0xAC sequence of bytes does not mean it should always treat it as a single character.

      Here are a few ways to enter UTF-8 encoded non-ASCII characters:

      • Use a UTF-8 aware editor (eg: VIM) and just type them in to string literals (or even variable names)
      • Use the Unicode escape sequence \x{XXXX} in a double quoted string (see above)
      • Read some data in from an XML file using a proper XML Parser module

      For more details see the utf8 man page.

[reply]
[d/l]
[select]
Re: detecting non-ascii chars in a string
by blahblahblah (Priest) on Dec 05, 2002 at 01:26 UTC
    I didn't have to look far into Jcode to find my answer -- only in the documentation. The getcode function will tell me what I need. I think as long as it returns ('ascii', 1) I can be sure the string contains only ascii chars.
[reply]
Re: detecting non-ascii chars in a string
by princepawn (Parson) on Dec 05, 2002 at 16:14 UTC
    The long answer is in this previous node

    Carter's compass: I know I'm on the right track when by deleting something, I'm adding functionality

[reply]

Back to Seekers of Perl Wisdom