Re: good way to implement utf8::is

sub is_utf8 {
    utf8::upgrade( $_[0] );
    return 1;
}
[download]

But seriously,

sub is_utf8 {
   my $s = "\x80" . $_[0];
   my $internal = unpack "p", pack "p", $s;
   return $s ne $internal;
}
[download]

Tested in 5.8.0, 5.8.8 and 5.10.0 using:

utf8::downgrade( my $empty_dn = ''          );  # 0
utf8::upgrade(   my $empty_up = ''          );  # 1
utf8::downgrade( my $ascii_dn = 'a'         );  # 0
utf8::upgrade(   my $ascii_up = 'a'         );  # 1
utf8::downgrade( my $hibit_dn = chr(0xC9)   );  # 0
utf8::upgrade(   my $hibit_up = chr(0xC9)   );  # 1
utf8::upgrade(   my $wide_up  = chr(0x2660) );  # 1

for (
    $empty_dn, $empty_up,
    $ascii_dn, $ascii_up,
    $hibit_dn, $hibit_up,
    $wide_up,
) {
   print is_utf8($_)?1:0, "\n";
}
[download]

Why do you need to know?

Update: Added test code.

Comment on Re: good way to implement utf8::is_utf8 for perl 5.8.0 Select or Download Code

Replies are listed 'Best First'.
Re^2: good way to implement utf8::is_utf8 for perl 5.8.0 by perl5ever (Pilgrim) on Mar 19, 2009 at 17:29 UTC
Thanks! Why do you need to know? I am maintaining an app that uses 5.8.0 and upgrading perl would require too much testing (it's a very big app.)	[reply]
Re^3: good way to implement utf8::is_utf8 for perl 5.8.0 by ikegami (Patriarch) on Mar 19, 2009 at 17:49 UTC
I meant: What use do you have for `is_utf8`? I find it's usually used where the following should be used: `utf8::downgrade($s, 1) or croak("Wide characters in foo argument");` [download]	[reply] [d/l] [select]
Re^4: good way to implement utf8::is_utf8 for perl 5.8.0 by perl5ever (Pilgrim) on Mar 19, 2009 at 18:24 UTC
Well, since you asked... I'm using it to debug a problem. I suspect using `utf::downgrade` is the solution, but I want to verify that. Specifically, I am using LWP to make https connections and LWP is calling Crypt::SSLeay. I'm trying to track down a problem which I think is being caused by the header having the UTF8 flag set (even though it contains only ASCII characters). The version of LWP I'm using is either very ancient or has been hacked -- it writes the header separately from the body. Newer versions of LWP concatenate the header and body before writing out the request. In my case this would immediately cause corruption when the body contained non-ASCII byte values. However, I'm only seeing corruption some of the time. My suspicion is that the corruption is occurring somewhere in the bowls of Crypt::SSLeay -- something like: `1. LWP syswrites out the header (with UTF8 flag set)is written 2. Crypt::SSLeay writes out encrypted header 3. LWP syswrites out the body 4. Crypt::SSLeay writes out the encrypted body but perhaps an internal + buffer that it uses has the UTF8 flag set which corrupts the body.` [download] The corruption is consistent with bytes with the high-bit set getting converted to their UTF8 encoding. I've noticed that the most recent of LWP performs a downgrade of the header to ensure its internal representation is bytes. Guess there must have been a good reason for that change...	[reply] [d/l] [select]
Re^5: good way to implement utf8::is_utf8 for perl 5.8.0 by ikegami (Patriarch) on Mar 19, 2009 at 20:15 UTC