in reply to Re: javascript encodeURI() -> perl uri_unescape_utf8() ?
in thread javascript encodeURI() -> perl uri_unescape_utf8() ?

You can see the characters by opening this simple script in your browser. (encodeURIComponent encodes a few more characters. So you better can see the difference) 
<script>
alert(encodeURIComponent('a å /'));
</script>

[download]
outputs: a%20%C3%A5%20%2F
The problem is that å is encoded into two "sequences": %C3%A5

Replies are listed 'Best First'.
Re^3: javascript encodeURI() -> perl uri_unescape_utf8() ?
by graff (Chancellor) on Dec 14, 2004 at 20:49 UTC
    The thing you are referring to as "two sequences" is actually the two-byte sequence for the utf-8 encoded character U00E5. (updated the grammar slightly to make more sense)

    Naturally, we'd love to have an elegant and concise way to interpret this correctly as utf8 text, but I don't know enough about the URI modules to provide much guidance in that direction.

    So instead, I'll offer an ad-hoc (but still somewhat concise) work-around -- it's a kluge, but it should work until you or some other monk can find the needed gems in the appropriate module(s): 

    use Encode;

# ... get the uri string into $_ by whatever means ...

$_ = "a%20%C3%A5%20%2F";

# first, let's turn the uri encoded string (with "%HH" for some bytes)
+ into binary:

s/\%([0-9a-f]{2})/chr(hex($1))/egi;

# then, since this produces a utf-8 byte sequence, let's "decode" that
+ into utf-8

$_ = decode( 'utf8', $_ );

# $_ now has utf8 flag set, and contains the string with expected unic
+ode characters

binmode STDOUT, ":utf8";

print;

    [download]
    The "binmode STDOUT" thing could be taken out if you add a "-CO" flag on the shebang line, I believe -- that "perlrun" option does the same thing as 'binmode STDOUT, ":utf8";'.
[reply]
[d/l]
      I think that CGI's 'unescape' method does the same thing. 
      use CGI qw(unescape);
$_ = "a%20%C3%A5%20%2F";
$value = CGI::unescape($_);

      [download]

      (it's in CGI/Util.pm)

[reply]
[d/l]
      Thanks for your suggestion.
      This does not solve my problem, unfortunately. I have found that the decode function in the Encode module should be able to do the job. 
      use Encode;
print decode('utf8', "\xC3\xA5");

      [download]
      This prints the character I need.
      The problem now is:
      How to go from %C3%A5 etc. to \xC3\xA5?
      I tried 
      use Encode;
$_='%C3%A5';
 s/%/\\x/g;
eval { $_=$_};
print decode('utf8', $_);

      [download]
      but it does not work as expected. :-)
      This was actually solved in the previous post by graff.
[reply]
[d/l]
[select]
        The problem now is: How to go from %C3%A5 etc. to \xC3\xA5?

        That's exactly what this line of code does: 

          s/\%([0-9A-Z]{2})/chr(hex($1))/egi;

        [download]
        Did you try that? The left side matches any "%HH" pattern, and captures the two hex digits into $1; in the right side, "hex()" interprets $1 as a hex number, and "chr()" turns that value into a single-byte character. (Or it should, unless maybe you've put a "use utf8" somewhere above this line -- if that's the problem, put this regex into its own block with "use bytes": 
        {
   use bytes;
   s/\%([0-9A-Z]{2})/chr(hex($1))/egi;
}

        [download]

        As for your eval attempt, I think the first "$_" in "$_=$_" needs to have a backslash in front of the dollar-sign: "\$_ = $_". In any case, I would advise against changing every percent-sign into "\x" -- there might be some cases of "%" that are not followed by a pair of hex digits.

        (update: Actually, in the context of dealing with uri strings, that last point is moot -- it surely must be the case in uri encoding that every "%" character is part of a "%HH" expression, and "%25" is used to express a literal percent-sign in the data. But the point is relevant for any data that might be a "defective" combination of uri encoding and plain text.)

[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom