Try turning it *off*. And, not to be snarky, but read the docs very carefully. You just have to.

Re-reading JSON::XS there was one key word I missed the first time around. Here is the relevant POD with my emphasis added.

$perl_scalar = decode_json $json_text

The opposite of encode_json: expects an UTF-8 (binary) string and tries to parse that as an UTF-8 encoded JSON text, returning the resulting reference. Croaks on error.

So I need to encode before passing it to decode_json(). Here is the working program

#!/usr/bin/perl

use v5.14;
use warnings;
use utf8::all;
use Encode;

use JSON::XS qw( decode_json );

my $wl = '{"creche": "crèche",
"¥": "£",
"₡": "волн"
}';

my $pattern_list = decode_json( encode("utf8", $wl) );
[download]

Hello.
There seems to be 2 problems.

One is JSON::XS expects 'encoded utf8' string as default, as you point out.
Second is utf8::all doesn't affect Slurp's io layer. When OP output to file and read it with Slurp's read_file, it is 'encoded utf8', not 'decoded utf8'. So the second example seems to succeed at a glance.


use strict;
use warnings;
use JSON::XS qw( decode_json );
use Data::Dumper;

binmode(STDOUT,":encoding(UTF-8)");

sub _p{return pack('U',$_[0])};

my ($wl,$pattern_list);
#create utf8 decoded(perl internal utf8) JSON character.
$wl = '{"creche": "cr'._p(0xE8).'che",';
$wl.= '"'._p(0xA5).'" : "'._p(0xA3).'",';
$wl.= '"'._p(8353).'": "'._p(1074)._p(1086)._p(1083)._p(1085).'"';
$wl.= '}';

#example 1 of OP
sub ex1 {
    my $pattern_list;

    #$pattern_list = decode_json($wl); #Wide character in subroutine e
+ntry

    #$pattern_list = JSON::XS->new->utf8(1)->decode($wl);#Wide charact
+er in subroutine ent

    #no warning: it seems this module expects encoded utf8 but decoded
+ utf8 by default
    $pattern_list = JSON::XS->new->utf8(0)->decode($wl);
}
#ex1();
#print Dumper $pattern_list;

#example 2
sub ex2 {
    use File::Slurp qw( read_file );
    use utf8::all;

    open my $fh, '>:encoding(UTF-8)', 'test_file2';
    print {$fh} $wl;
    close $fh;

    #here utf8::all failed to set Slurp's io layer
    my $buffer= read_file('test_file2');
    print utf8::is_utf8($buffer) ? "buffer:utf8 flagged\n" : "buffer:n
+ot utf8 flagged\n";

    #you get 'encoded utf8 bytes and that is default for JSON::XS
    $pattern_list = decode_json( $buffer);

    #pattern_list is encoded utf8 string, not decoded
    print utf8::is_utf8($pattern_list) ? "pattern:utf8 flagged\n" : "p
+attern:not utf8 flag

}
ex2();
print Dumper $pattern_list;
[download]

JSON::XS's utf8 seems to me very different from other modules like DBD:: modules, Template's binmode option.

You are trying to write a unicode encoded character to a file but the file has not been opened with an encoding that can cope. You need something like this (untested):

 {open(my $F, ">:encoding(UTF-8)", $f) or die "Cannot open $f";
  say {$F} $s;
 }
[download]

I am using utf8::all which already takes care of turning on the different unicode parts.

The errors do not come from reading or writing a file. The error is caused by using decode_json() on a string which should work.

Thanks for the pointer to utf8::all, it'll be handy.

    -- Chip Salzenberg, Free-Floating Agent of Chaos

Could you replace the characters in the input to JSON with a's one by one until the offending wide character disappears? We will then know exactly which input character is causing the problem. Thanks.

Neither JSON::XS nor the natvie JSON parsers in Javascript execute Javascript code. They all (should) parse the text and reconstruct the data structure using a JSON parser, not the Javascript eval statement, exactly for the reason of not allowing easy Javascript code execution within the page context.

The web application should support that by sending the appropriate content type - which is application/json, at least according to RFC 4627.

Using JSONP sacrifices that security for the convenience of circumventing the same origin policy.

Indeed, yes. No-one in their right mind would actively and intentionally execute passed code. Maybe I'm paranoid, but I won't have executable code coming in from the network under any circumstances. Before I'll accept it, it must be not capable of being executed. Accepting executable code and then trusting that it will never be called... erm... Just seems to me that if you were going to attack a system you would look very closely at JSON and related parsing mechanisms because code is, by definition, already accepted. You're already half way there. In fact you have been supplied with a ready built framework for injecting your malice. After all, if it isn't runnable code it isn't JSON and PHP plus JSON seems like a perfect storm. It's just not likely to be failsafe. At least Perl can be made failsafe.