isync has asked for the wisdom of the Perl Monks concerning the following question:

I think I found an inconsistency between JSON::XS' and JSON's to_json routine: JSON::XS produces utf-8 which can be utf8::decode()ed, while JSON does not.
Now I'm puzzled, as I don't see what I'm doing wrong. (Is JSON::XS fixing something automagically, or getting corner-cases right, and JSON is not?). Demo script follows: 
#!/usr/bin/env perl

# http://www.perl.com/pub/2012/04/perlunicook-standard-preamble.html
use utf8;      # so literals and identifiers can be in UTF-8
use v5.12;     # or later to get "unicode_strings" feature
use strict;    # quote strings, declare variables
use warnings;  # on by default
use warnings  qw(FATAL utf8);    # fatalize encoding glitches
use open      qw(:std :utf8);    # undeclared streams in UTF-8
use charnames qw(:full :short);  # unneeded in v5.16

use Dancer;

use Encode ();
use HTML::Entities ();
use JSON::XS ();
use Devel::Peek;
use utf8;

our $string = "Hello – World";

get '/' => sub {
    return '<a href="/ok">ok</a> <a href="/crash">crash</a>';
};

get '/ok' => sub {
    my $perl_internal = Encode::decode('utf-8', $string);    # "use ut
+f8;" above made $string be in utf-8, decode to Perl-internal (= Unico
+de, stored internally as utf-8, right?)
    HTML::Entities::decode_entities($perl_internal);    # in void cont
+ext, expand entities in-place to Unicode
    my $json = JSON::XS::encode_json([$perl_internal]);    # pass a Pe
+rl data structure, and expect output to be a utf-8 encoded, binary st
+ring

    Dump( $perl_internal );
    Dump( $json );

    my $ok = utf8::decode($json);                # fairly good way to 
+validate utf-8

    return "JSON:$json <br>". ($ok ? "ok" : "not ok");
};

get '/crash' => sub {
    my $perl_internal = Encode::decode('utf-8', $string);
    HTML::Entities::decode_entities($perl_internal);
    my $json = to_json([$perl_internal],{pretty => 0});    # JSON's to
+_json, in contrary to JSON::XS's to_json seems to do something wrong,

    Dump( $perl_internal );
    Dump( $json );
                                # or is less clever in automagically h
+andling what I'm doing wrong while preparing the scalar
    my $ok = utf8::decode($json);

    return "JSON:$json <br>". ($ok ? "ok" : "not ok");
};
 
dance;

[download]
$ perl -v
This is perl 5, version 18, subversion 2 (v5.18.2) built for x86_64-linux-gnu-thread-multi

JSON::XS: 2.34
JSON: 2.61, now updated to 2.90 - no change
... both a little outdated

Replies are listed 'Best First'.
Re: JSON::XS produces valid utf-8, and JSON doesn't - why?
by Anonymous Monk on Nov 12, 2015 at 23:54 UTC

    Some remarks:

    my $ok = utf8::decode($json); is not the best way to validate utf8. Because:
    1. It only validates Perl's notion of utf-8. Which is not exactly what the Unicode Consortium says. 
      use feature 'say';
binmode STDOUT, 'encoding(utf-8)';

my $str = "\xFC\x90\x80\x80\x80\x80";
my $ok = utf8::decode( $str );

say $ok ? 'ok' : 'not ok';
say $str;

      [download]
      output: 
      ok
Code point 0x10000000 is not Unicode, may not be portable at demo.pl l
+ine 8.
"\x{10000000}" does not map to utf8 at demo.pl line 8.
\x{10000000}

      [download]
    2. It always decodes a string in place. So doing 
      my $ok = utf8::decode($json);
Dump( $json );

      [download]
      is not super useful
    Second, according to JSON's docs, encode_json is equivalent to JSON->new->utf8->encode($perl_scalar), while to_json is JSON->new->encode($perl_scalar). I don't know how these modules work, but the presence/absence of ->utf8 probably makes some difference?
[reply]
[d/l]
[select]
Re: Is JSON::XS doing more things right here,
by Laurent_R (Canon) on Nov 12, 2015 at 18:49 UTC
    Can you perhaps describe what the inconsistency is?
[reply]
      post updated: JSON::XS produces "valid" utf-8, and JSON does not.
[reply]

        "does not" is one of most vague problem descriptions; right up there with "doesn't work".

        A Perl module that produces JSON has two valid choices of what to output: 1) A UTF-8 string that has been tagged as being UTF-8 (from Perl's perspective). 2) A UTF-8 string encoded as a sequence of bytes from Perl's perspective [same as (1) just without the "is utf-8" flag set].

        Your code presumes only one of those possibilities. I know that JSON::XS defaults to returning (1) but can be asked to return (2). I haven't looked into what JSON::PP gives.

        When dealing with UTF-8 problems in Perl, it is best to use Devel::Peek.

        - tye        

[reply]
Re: JSON::XS produces valid utf-8, and JSON doesn't - why?
by isync (Hermit) on Nov 13, 2015 at 01:45 UTC
    Aha, now Devel::Peek output has some evidence:
    Read more... (737 Bytes)
[reply]

Back to Seekers of Perl Wisdom