John M. Dlugosz has asked for the wisdom of the Perl Monks concerning the following question:

Say I have a string that contains UTF-8 encoded text, but Perl thinks that string has byte orientation. How do I set the magic bit to tell Perl that it's character oriented?

I did it before with a taint-like trick, but I can't remember how exactly.

It seems to work if I use a taint-like trick, in that /^(.*)$/so; will give me $1 in character-orientation if this RE was issued while use utf8 was in effect, regardless of the orientation of the string that the RE was applied to.

Is that right, or just luck?

Is there a better/proper way to do it, yet?

It's maddening that features of Unicode support were not finished (e.g. I/O disciplines) and newer versions do more and more "character" (not byte) stuff, but the primitives needed to fill the gap ourselves are not exposed, for no good reason.

—John

Edit kudra, 2001-07-23 Corrected title

Replies are listed 'Best First'.
Re: How do you impose
by wog (Curate) on Jul 22, 2001 at 09:48 UTC

    I think you can use pack and unpack to accomplish this:

    my $utf8; { local $^W; # warnings are thrown when unpacking # invalid UTF8, ignore them, we'll check # for problems be seeing if the string is # null at the end and shouldn't be. $utf8 = pack "U*", unpack "U*", $raw; } deal_with_malformed_utf8() if length $utf8 == 0 and length $raw != 0;

    If you are using a very recent version (i.e. experimental) of perl you can use the Encode module's decode_utf8 function, or use the internal function utf8::decode which it is implemented with:

    use Encode; my $string = decode_utf8($bytes); deal_with_malformed_utf8() unless defined $string # or utf8::decode($string = $bytes) # of course, you can just or deal_with_malformed_utf8();# use one variable, and not # keep the raw bytes.

    (update: (Because it may seem odd:) The rational behind the decode in the function names is that these functions "[decode] from UTF-8 to a sequence of logical characters." )</o>

    Also in the experimental version of perl the Encode module provides a _utf8_on function that will turn on that flag which says "this value carries UTF-8", however the documentation advises that this function is internal and may change, and that one must be very sure you are dealing with valid UTF-8 first.

      Ah, so unpack "U" will pull a multibyte character, ignoring the char/byte orientation on the string being scanned. That makes sense. I was thinking unpack using C and pack using U, or something along those lines, but could not see how unpack/pack could work.

      In current Perl, I think the taint-like trick is faster than having to pull and repack all the chars in the string.

      I look forward to the new features when v5.8 comes out. Thanks for pointing that out. It's very annoying that there's not a set and test function for the mode bit, especially in the even earlier days when not all the built-ins worked right yet. Even today without input disciplines, we need to set the charmode on everything read in from a file!

title was truncated
by John M. Dlugosz (Monsignor) on Jul 22, 2001 at 10:23 UTC
    I think the intermittent "quote" bug bit me, since my title was severly truncated. It originally said something like How do you impose "character" orientation on a string? if an editor would be so kind as to fix that.