Re: Re: How to know if a string has UTF-8?

The advantage of your function is to work with Perl-5.6 too, since it check for group of bytes. But since that's no meaning to check the UTF-8 on Perl without support, since the user won't be able to generate and work with true UTF-8 data, I made that (please check if it's ok):

sub _is_unicode {
  if ($] >= 5.8) {
    eval(q`
      if ( $data =~ /[\x{100}-\x{10FFFF}]/s) { return 1 ;}}
    `);
  }
  else {
    ## No Perl support for UTF-8! ;-/
    return undef ;
  }

  return undef ;
}
[download]

I made this based in table at section #Unicode_Encodings in this POD: http://search.cpan.org/author/JHI/perl-5.8.0/pod/perlunicode.pod

Note that when the user isn't in Perl 5.8 I don't want, actually can't, handle the UTF-8, since the implementation was there only to handle UTF-8 as a group of bytes, not as characters. Note that for Perl 5.7 (that isn't stable) I won't hanbdle UTF-8 too.

Now I have a big problem. All the computer that I have here are based on ASCII-extended (ISO-8859-1), and for us is just what we need. But is impossible to tell that you made Unicode support without really test it on UTF-8 platforms (EBCDIC), like a Japanese PC. So, any one have some idea, or a Japanese PC?

Graciliano M. P.
"The creativity is the expression of the liberty".

Comment on Re: Re: How to know if a string has UTF-8? Download Code

Replies are listed 'Best First'.
Re: Re: Re: How to know if a string has UTF-8? by graff (Chancellor) on May 28, 2003 at 22:10 UTC
I think the strategy for your "_is_unicode" sub should be fine (I haven't tried it yet, but probably will...) -- it has the nice feature of not requiring the Encode module (important if the script might be run with older versions of Perl), and handles both validation and "presence of non-ascii" questions in one swoop. However, I wonder if the condition inside the eval should be: `if ( $date =~ /[\x{0080}-\x{FFFF}]/ ) {return 1;}` [download] Note that the "Latin1" characters (x0080-x00ff) are encoded using two bytes in utf8, and are officially "non-ascii". If you want to pass them through (because your current system can handle Latin1), you still need to convert them to iso-8859-1 (i.e. single-byte encoding) in order for your system to display them correctly -- back to the Encode module... (And I still don't think you need to worry about 32-bit code points -- not for a while.) As for providing an intelligible view of utf8 data on a single-byte Latin1 system, you could check out Text::Unidecode -- though it might not cover all languages in the manner you would prefer. Another thing to try would be a unicode/utf8-aware browser. There is also a free utility called "yudit" that runs on unix/linux and (I think) ms-windows too -- it's a utf8-aware text editor.	[reply] [d/l]
Re: Re: Re: Re: How to know if a string has UTF-8? by gmpassos (Priest) on May 29, 2003 at 05:44 UTC
Thanks for the advices. But you said: ... it has the nice feature of not requiring the Encode module (important if the script might be run with older versions of Perl),... Note that I can't use this REGEX on Perl-5.6.1, since the range of \x is much bigger than 255! This is why I put it inside eval(), or it won't compile. Graciliano M. P. "The creativity is the expression of the liberty".	[reply]