in reply to JSON and utf8 flag

For a quick solution:
> diff -u XS.xs_orig XS.xs --- XS.xs_orig 2017-09-05 13:30:07.933454000 +0300 +++ XS.xs 2017-09-06 08:35:35.224749000 +0300 @@ -1214,7 +1214,7 @@ SvPOK_only (sv); *SvEND (sv) = 0; - if (utf8) + if (utf8 && !(dec->json.flags & F_LATIN1)) SvUTF8_on (sv); } else

Replies are listed 'Best First'.
Re^2: JSON and utf8 flag
by ikegami (Patriarch) on Sep 05, 2017 at 17:12 UTC

    Don't break JSON::XS because of a bug in your code!

      Where?

      use Devel::Peek; use JSON::XS; my $smile = "☻"; my $j = JSON::XS->new->latin1(1)->encode(["$smile"]); my $d = JSON::XS->new->utf8(1)->latin1(1)->decode($j); Dump $$d[0];
      Output:
      SV = PV(0x8f19788) at 0x8f323bc REFCNT = 1 FLAGS = (POK,pPOK,UTF8) PV = 0x8f4be68 "\342\230\273"\0 [UTF8 "\x{263b}"] CUR = 3 LEN = 10

      I put data without UTF8 flag to JSON and I want to get data without UTF8 flag from this JSON.

      How to do it?

        > I put data without UTF8 flag to JSON

        RFC7159:

        JSON text SHALL be encoded in UTF-8, UTF-16, or UTF-32.

        ($q=q:Sq=~/;[c](.)(.)/;chr(-||-|5+lengthSq)`"S|oS2"`map{chr |+ord }map{substrSq`S_+|`|}3E|-|`7**2-3:)=~y+S|`+$1,++print+eval$q,q,a,

        First of all, the UTF8 flag is irrelevant. I believe you are actually complaining that $smile ne $d->[0].


        Secondly, you claim your source code includes the following:

        my $smile = "☻";
        

        That's not possible unless you have use utf8;. You actually have the following:

        my $s_orig = "\xE2\x98\xBB";

        If interpreted as Unicode Code Points (as ->encode does), you have LATIN SMALL LETTER A WITH CIRCUMFLEX, followed by START OF STRING, followed by RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK.


        Finally, how to round trip:

        use Test::More tests => 1; use JSON::XS qw( decode_json encode_json ); my $s_orig = "\xE2\x98\xBB"; my $data_orig = [ $s_orig ]; my $json_utf8 = encode_json($data_orig); my $data_got = decode_json($json_utf8); my $s_got = $data_got->[0]; is($s_got, $s_orig);

        If you actually want to store a smile,

        use Test::More tests => 1; use JSON::XS qw( decode_json encode_json ); my $smile_utf8 = "\xE2\x98\xBB"; utf8::decode( my $s_orig = my $smile_utf8 ); my $data_orig = [ $s_orig ]; my $json_utf8 = encode_json($data_orig); my $data_got = decode_json($json_utf8); my $s_got = $data_got->[0]; is($s_got, $s_orig);