in reply to How to know if a string has UTF-8?

Hi again, gmpassos :)

This is what I use: (I didn't write this myself)

# ValidUTF8 came from: http://people.netscape.com/ftang/utf8/isutf8.pl
sub ValidUTF8 {

        local ( $utf8 ) = pop (@_);

        if($utf8 =~ /^(([\0-\x7F])|([\xC0-\xDF][\x80-\xBF])|([\xE0-\xE
+F][\x80-\xBF][\x80-\xBF])|([\xF0-\xF7][\x80-\xBF][\x80-\xBF][\x80-\xB
+F])|([\xF8-\xFB][\x80-\xBF][\x80-\xBF][\x80-\xBF][\x80-\xBF])|([\xFC-
+\xFE][\x80-\xBF][\x80-\xBF][\x80-\xBF][\x80-\xBF][\x80-\xBF]))*$/)
        {
                return ! ($utf8 =~ /([\xC0-\xC1])|([\xE0][\x80-\x9F])|
+([\xF0][\x80-\x8F])|([\xF8][\x80-\x87])|([\xFC][\x80-\x83])/);
        }
        else
        {
                return 0;
        }
}

[download]

Replies are listed 'Best First'.
Re: Re: How to know if a string has UTF-8?
by gmpassos (Priest) on May 28, 2003 at 18:08 UTC
    The advantage of your function is to work with Perl-5.6 too, since it check for group of bytes. But since that's no meaning to check the UTF-8 on Perl without support, since the user won't be able to generate and work with true UTF-8 data, I made that (please check if it's ok): 
    sub _is_unicode {
  if ($] >= 5.8) {
    eval(q`
      if ( $data =~ /[\x{100}-\x{10FFFF}]/s) { return 1 ;}}
    `);
  }
  else {
    ## No Perl support for UTF-8! ;-/
    return undef ;
  }

  return undef ;
}

    [download]
    I made this based in table at section #Unicode_Encodings in this POD: http://search.cpan.org/author/JHI/perl-5.8.0/pod/perlunicode.pod

    Note that when the user isn't in Perl 5.8 I don't want, actually can't, handle the UTF-8, since the implementation was there only to handle UTF-8 as a group of bytes, not as characters. Note that for Perl 5.7 (that isn't stable) I won't hanbdle UTF-8 too.

    Now I have a big problem. All the computer that I have here are based on ASCII-extended (ISO-8859-1), and for us is just what we need. But is impossible to tell that you made Unicode support without really test it on UTF-8 platforms (EBCDIC), like a Japanese PC. So, any one have some idea, or a Japanese PC?

    Graciliano M. P.
    "The creativity is the expression of the liberty".

[reply]
[d/l]
      I think the strategy for your "_is_unicode" sub should be fine (I haven't tried it yet, but probably will...) -- it has the nice feature of not requiring the Encode module (important if the script might be run with older versions of Perl), and handles both validation and "presence of non-ascii" questions in one swoop. However, I wonder if the condition inside the eval should be: 
        if ( $date =~ /[\x{0080}-\x{FFFF}]/ ) {return 1;}

      [download]
      Note that the "Latin1" characters (x0080-x00ff) are encoded using two bytes in utf8, and are officially "non-ascii". If you want to pass them through (because your current system can handle Latin1), you still need to convert them to iso-8859-1 (i.e. single-byte encoding) in order for your system to display them correctly -- back to the Encode module...

      (And I still don't think you need to worry about 32-bit code points -- not for a while.)

      As for providing an intelligible view of utf8 data on a single-byte Latin1 system, you could check out Text::Unidecode -- though it might not cover all languages in the manner you would prefer. Another thing to try would be a unicode/utf8-aware browser. There is also a free utility called "yudit" that runs on unix/linux and (I think) ms-windows too -- it's a utf8-aware text editor.

[reply]
[d/l]
        Thanks for the advices. But you said:

        ... it has the nice feature of not requiring the Encode module (important if the script might be run with older versions of Perl),...

        Note that I can't use this REGEX on Perl-5.6.1, since the range of \x is much bigger than 255! This is why I put it inside eval(), or it won't compile.

        Graciliano M. P.
        "The creativity is the expression of the liberty".

[reply]

In Section Seekers of Perl Wisdom