In one respect, it may be that 5.10's behavior is "consistent" in a way that 5.8's behavior is not:

# in perl 5.8.8:
perl -Mencoding=utf8 -e '$a="\x51";$b="\xE1"; printf( "a=%x b=%x\n",or
+d($a),ord($b))'
a=51 b=fffd

perl -Mencoding=utf8 -e '$a=chr(hex("51"));$b=chr(hex("E1")); printf( 
+"a=%x b=%x\n",ord($a),ord($b))'
a=51 b=e1
[download]

While it's probably true that maintaining your own local version of CGI::Util will be "easier", it might still make sense to consider whether there's a better way to handle this issue:

I cannot easily remove the 'use encoding "utf8"' from my script as the pragma does "some magic" that prevents encoding-related disasters (double-encoded strings) further down the road.

Figuring what that "magic" is and where it applies in your code, and then looking for better ways to achieve the same result, might be better in the long run. In particular, since "use encoding" is not scoped, replacing that solution with some other (scoped and/or focused) approach for your encoding-related disasters would seem prudent and worthwhile.

Thanks for your example. I understand that "use encoding" applies to the program source (string literals etc.) but that it affects the chr function was quite a surprise to me.

As for replacing it with something more manageable, I certainly will, but I needed a workaround right now and could not afford to dig into the code at the moment.

From perldoc perlunicode:

The "chr()" and "ord()" functions work on characters, similar to "pack("W")" and "unpack("W")", not "pack("C")" and "unpack("C")". "pack("C")" and "unpack("C")" are methods for emulating byte-oriented "chr()" and "ord()" on Unicode strings. While these methods reveal the internal encoding of Unicode strings, that is not something one normally needs to care about at all.

Being a no-expert at all in that, just hope the following can help to give you or others a good direction:

perl -Mencoding=utf8 -le 'print unpack "C", chr 156'
156

perl -Mencoding=utf8 -M'Encode qw(from_to)' -le '$c = chr 156; from_to($c, "iso-8859-3", "utf-8"); print ord $c'
156

Btw, more or less off-topic, CGI::Util can unescape unicode with the uXXXX convention but does not follow this convension when doing escaping -; i.e. I changed its encoding routine from

$toencode=~s/([^a-zA-Z0-9_.~-])/uc sprintf("%%%02x",ord($1))/eg;
to

$toencode =~ s/([^a-zA-Z0-9_.-])/(ord($1) > 255) ? sprintf("%%u%04X", ord($1)) : sprintf("%%%02X", ord($1))/eg;

Thanks, it hasn't bitten me yet but I'll keep an eye on it. I have encountered problems in the other half of the expression.

A form field value of '<ę' (less than sign, small e with ogonek) got urlencoded as '%3C%C4%99' by the browser and decoded as '<\x{fffd}\x{fffd}' by 'chr(hex($1))'. What really drove me mad was that 'ę' itself ('%C4%99') was handled properly so I thought it was some HTML escaping misfeature that appeared between CGI versions.

You've got another problem. Your browser incorrectly passed the ę character as %C4%88 that is, as a string which happens to be UTF-8 but should have passed it as %u0119. You've got to *guess* during your URL parsing if someone has started passing you a byte-string which happens to look like a UTF-8 string and if so, reinterpret it as UTF-8 instead. If you're being fully careful, you should also be using the Content-Type of the HTTP header your server sends, the client sends, and the HTML content encoding you sent when generating your page.

Here's what I use.

sub uri_unescape {
    my $to_decode = shift;

    return if ! defined $to_decode;
    $to_decode =~ s/\+/ /g;

    # A "good enough" check to see if the browser encoded high bit
    # characters as UTF-8 by looking for something that resembles a
    # URL-encoded utf8 octet series
    #
    # utf8 encoding starts with 11xxxxxx in the first byte and
    # 10xxxxxx in the subsequent bytes
    # http://en.wikipedia.org/wiki/UTF-8#Description
    my $is_byte_encoded_utf8 = $to_decode =~ /%[c-fC-F][[:xdigit:]]%[8
+9abAB][[:xdigit:]]/;

    $to_decode =~ s/%([[:xdigit:]]{2})/chr hex "0x$1"/eg;

    # Decode the "UTF-8" if it looked like it was such. If I enter
    # this block, the string may now be UTF-8 encoded as a perl
    # string.
    if ($is_byte_encoded_utf8) {
        $to_decode = Encode::decode( 'utf8', $to_decode );
    }

    # Promote %uXXXX escapes up to characters. This is after the
    # Encode call so I don't inadvertently cause a promotion to utf8
    # and then ask Encode to do another promotion.
    $to_decode =~ s/%u([[:xdigit:]]{4})/chr hex "0x$1"/eg;

    return $to_decode;
}
[download]

⠤⠤ ⠙⠊⠕⠞⠁⠇⠑⠧⠊

perl -Mutf8 -le "print ord chr 156"
[download]