Re^5: JSON::XS (and JSON::PP) appear to generate invalid UTF-8 for character in range 127 to 255

Damn right I disagree. It is not Perl's problem that someone using a function that's documented to check for accidental double-encoding to check if something is valid UTF-8. That's akin to using uc to get the first character of a string. There's nothing Perl can do to stop you from using a function completely unrelated to the one you want to use.

Except it's kind of hard to understand what the heck the function is doing. 'flagged as utf8', 'store a string internally'... too many implementation details. Do you expect many people to understand it?

This is the second time this thread you've implied that I maintain that Perl's handling of UTF-8 isn't confusing. That's a lie. The former bugs in Perl (some still present) and the plethora of buggy XS module (because XS is hard!) has led people like you to disseminate misinformation, which has created a self-feeding vicious loop of confused people. I've repeatedly said that Perl should be able to differentiate encoded strings from decoded strings and prevent you from mixing them.

Maybe you missed that, ikegami... but I actually never have any problems with mojibake in my Perl code... unlike some other people. I know where these kinds of bugs come from and how to fix them. Works for me, eh?

Speaking of misinformation, improper upgrading doesn't cause double-encoding. Quite the opposite, it causes a string encoded using UTF-8 to become decoded. (Upgrading a strings that isn't encoded using UTF-8 creates a corrupt scalar. perl -MDevel::Peek -MEncode=_utf8_on -we"$_ = qq{\x80}; _utf8_on($_); Dump($_)")

I've called it 'upgrading' (in quotes) in honor of utf8::upgrade (perl -MDevel::Peek -CO -E 'my $s = "\xff"; Dump $s; say $s; utf8::upgrade($s); Dump $s; say $s' - note that I don't care one bit how Perl actually does that). Not sure why you even mentioned _utf8_on. Anyway, I really dislike this term 'double encoding', because that implies that the problem is with the encoding, and not the decoding part (encoding needs some decoding first). Why isn't double encoding utf-8 a no-op? Really, just explain it in your own words.

(perl -MEncode=encode -E 'say encode("Latin-1", encode("Latin-1", "\xff")) doesn't seem to do much of anything?)

Comment on Re^5: JSON::XS (and JSON::PP) appear to generate invalid UTF-8 for character in range 127 to 255 Select or Download Code

Replies are listed 'Best First'.
Re^6: JSON::XS (and JSON::PP) appear to generate invalid UTF-8 for character in range 127 to 255 by ikegami (Patriarch) on Dec 07, 2014 at 15:33 UTC
Except it's kind of hard to understand what the heck the function is doing. 'flagged as utf8', 'store a string internally'... too many implementation details. This is my very problem with you: You bring up internal details for no reason. And these implementation details just end up confusing people, not helping them. Except it's kind of hard to understand what the heck the function is doing That a module is badly documented is not Perl's fault. Maybe you missed that, ikegami... but I actually never have any problems with mojibake in my Perl code... Yeah, I know you know you know better. I've called it 'upgrading' (in quotes) in honor of utf8::upgrade That doesn't double encode either. That doesn't change the string at all. (Remove the upgrade from your code and you get the same output.) Not sure why you even mentioned _utf8_on `_utf8_on` and `utf8::upgrade` both end up with an upgraded string, `_utf8_one` is the one used throughout the docs for Test::utf8, and your comment was wrong whichever function you were talking about.	[reply] [d/l] [select]
Re^7: JSON::XS (and JSON::PP) appear to generate invalid UTF-8 for character in range 127 to 255 by Anonymous Monk on Dec 07, 2014 at 22:50 UTC
That doesn't double encode either. That doesn't change the string at all. (Remove the upgrade from your code and you get the same output.) LOL you just won't budge! Now, why doesn't `utf8::upgrade` 'change' the string if it does the following: Converts in-place the internal representation of the string from an octet sequence in the native encoding (Latin-1 or EBCDIC) to UTF-X Am I to assume that it won't 'change' the string only if the string is indeed in 'native encoding', which can be Latin-1 or EBCDIC? (but not, say, UTF-8?) That doesn't seem terribly confusing. Yes, it doesn't do double encoding, as far as I can tell, it looks more like single decoding (`perl -MEncode=decode -MDevel::Peek -E 'my $s = "\xff"; Dump decode("Latin-1", $s); utf8::upgrade($s); Dump $s'`). _utf8_on and utf8::upgrade both end up with an upgraded string, _utf8_one is the one used throughout the docs for Test::utf8, and your comment was wrong whichever function you were talking about. And `utf8::upgrade` is the one that produces exactly the kind of strings that `is_sane_utf8` is intended to catch, so 'upgraded' strings is a good enough description ('decoded from Latin-1' is also good), unlike 'double encoding', where, for `is_sane_utf8` purposes, the problem is neither with encoding, nor does something needs to happen twice.	[reply] [d/l] [select]
Re^8: JSON::XS (and JSON::PP) appear to generate invalid UTF-8 for character in range 127 to 255 by ikegami (Patriarch) on Dec 10, 2014 at 07:30 UTC
why doesn't utf8::upgrade 'change' the string if it does the following: Because it's meant not to, and none of that changes the string. And utf8::upgrade is the one that produces exactly the kind of strings that is_sane_utf8 is intended to catch, so 'upgraded' strings is a good enough description Nope. It only flags some upgraded strings. unlike 'double encoding', Yeah, saying it checks for that would be wrong too. I just took your word for it earlier.	[reply]