Btw, more or less off-topic, CGI::Util can unescape unicode with the uXXXX convention but does not follow this convension when doing escaping -; i.e. I changed its encoding routine from
$toencode=~s/([^a-zA-Z0-9_.~-])/uc sprintf("%%%02x",ord($1))/eg;
to
$toencode =~ s/([^a-zA-Z0-9_.-])/(ord($1) > 255) ? sprintf("%%u%04X", ord($1)) : sprintf("%%%02X", ord($1))/eg;
| [reply]
[d/l]
[select] |
Thanks, it hasn't bitten me yet but I'll keep an eye on it. I have encountered problems in the other half of the expression.
A form field value of '<ę' (less than sign, small e with ogonek) got urlencoded as '%3C%C4%99' by the browser and decoded as '<\x{fffd}\x{fffd}' by 'chr(hex($1))'. What really drove me mad was that 'ę' itself ('%C4%99') was handled properly so I thought it was some HTML escaping misfeature that appeared between CGI versions.
| [reply] |
You've got another problem. Your browser incorrectly passed the ę character as %C4%88 that is, as a string which happens to be UTF-8 but should have passed it as %u0119. You've got to *guess* during your URL parsing if someone has started passing you a byte-string which happens to look like a UTF-8 string and if so, reinterpret it as UTF-8 instead. If you're being fully careful, you should also be using the Content-Type of the HTTP header your server sends, the client sends, and the HTML content encoding you sent when generating your page.
Here's what I use.
sub uri_unescape {
my $to_decode = shift;
return if ! defined $to_decode;
$to_decode =~ s/\+/ /g;
# A "good enough" check to see if the browser encoded high bit
# characters as UTF-8 by looking for something that resembles a
# URL-encoded utf8 octet series
#
# utf8 encoding starts with 11xxxxxx in the first byte and
# 10xxxxxx in the subsequent bytes
# http://en.wikipedia.org/wiki/UTF-8#Description
my $is_byte_encoded_utf8 = $to_decode =~ /%[c-fC-F][[:xdigit:]]%[8
+9abAB][[:xdigit:]]/;
$to_decode =~ s/%([[:xdigit:]]{2})/chr hex "0x$1"/eg;
# Decode the "UTF-8" if it looked like it was such. If I enter
# this block, the string may now be UTF-8 encoded as a perl
# string.
if ($is_byte_encoded_utf8) {
$to_decode = Encode::decode( 'utf8', $to_decode );
}
# Promote %uXXXX escapes up to characters. This is after the
# Encode call so I don't inadvertently cause a promotion to utf8
# and then ask Encode to do another promotion.
$to_decode =~ s/%u([[:xdigit:]]{4})/chr hex "0x$1"/eg;
return $to_decode;
}
| [reply]
[d/l] |