in reply to Re: Re: How to know if a string has UTF-8?
in thread How to know if a string has UTF-8?

I think the strategy for your "_is_unicode" sub should be fine (I haven't tried it yet, but probably will...) -- it has the nice feature of not requiring the Encode module (important if the script might be run with older versions of Perl), and handles both validation and "presence of non-ascii" questions in one swoop. However, I wonder if the condition inside the eval should be:
if ( $date =~ /[\x{0080}-\x{FFFF}]/ ) {return 1;}
Note that the "Latin1" characters (x0080-x00ff) are encoded using two bytes in utf8, and are officially "non-ascii". If you want to pass them through (because your current system can handle Latin1), you still need to convert them to iso-8859-1 (i.e. single-byte encoding) in order for your system to display them correctly -- back to the Encode module...

(And I still don't think you need to worry about 32-bit code points -- not for a while.)

As for providing an intelligible view of utf8 data on a single-byte Latin1 system, you could check out Text::Unidecode -- though it might not cover all languages in the manner you would prefer. Another thing to try would be a unicode/utf8-aware browser. There is also a free utility called "yudit" that runs on unix/linux and (I think) ms-windows too -- it's a utf8-aware text editor.

Replies are listed 'Best First'.
Re: Re: Re: Re: How to know if a string has UTF-8?
by gmpassos (Priest) on May 29, 2003 at 05:44 UTC
    Thanks for the advices. But you said:

    ... it has the nice feature of not requiring the Encode module (important if the script might be run with older versions of Perl),...

    Note that I can't use this REGEX on Perl-5.6.1, since the range of \x is much bigger than 255! This is why I put it inside eval(), or it won't compile.

    Graciliano M. P.
    "The creativity is the expression of the liberty".