kimlid2810 has asked for the wisdom of the Perl Monks concerning the following question:

hello all. i have this script: 

#!/usr/bin/perl

use LWP::Simple;                # From CPAN
use JSON qw( decode_json );     # From CPAN
use Data::Dumper;               # Perl core module
use strict;                     # Good practice
use warnings;                   # Good practice

my $trendsurl = "http://www.somesite.com/";

my $json = get( $trendsurl );
die "Could not get $trendsurl!" unless defined $json;

open my $file_write, '>', '/tmp/jsonParser';
print $file_write $json;
close($file_write);

open my $file_read, '<', '/tmp/jsonParser';
$json = <$file_read>;

my $decoded_json = decode_json( $json );

print Dumper $decoded_json;

[download]
so here is my problem... i use linux fedora 19. this link contains both english and greek characters. its metatag says: 
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">

[download]
i downloaded this link with wget and both english and greek characters were readable. i downloaded this link with wget and saved it to a file. the file command for this file returns this: 
file: UTF-8 Unicode text, with very long lines, with no line terminato
+rs

[download]
and its characters are readable. the $json variable when printed in screen, prints all the characters as readable. the $json variable when saved to a file like i do above, produces a file with readable characters and this file command response: 
file: UTF-8 Unicode text, with very long lines, with no line terminato
+rs

[download]
when i open this file with perl and reassign its contents to the $json variable and print it to screen, all the characters are readable. finally when i decode $json variable the greek characters are transformed to this: 
"\x{3a3}\x{391}\x{39b}\x{3a4}\x{3a3}\x{39c}\x{3a0}\x{39f}\x{3a5}\x{3a1
+}\x{393}\x{39a}"

[download]
the entire response from the site, is json valid, checked with JSONLint. so i must be doing something wrong with the json parser. despite my googling, i couldn't find some hope for a solution. i would welcome any ideas and i would be eternally grateful to anyone providing a possible solution or just a faint hint. thank you :)

Replies are listed 'Best First'.
Re: parsing json, problem with encodings
by Your Mother (Archbishop) on Mar 20, 2014 at 19:01 UTC

    Looks okay, I think. Just being escaped by the dumping code.

    ~>perl -CSD -le'print "\x{3a3}\x{391}\x{39b}\x{3a4}\x{3a3}\x{39c}\x{3a0}\x{39f}\x{3a5}\x{3a1}\x{393}\x{39a}"' 
    ΣΑΛΤΣΜΠΟΥΡΓΚ

    You might try printing out of the perl variables directly, for example print $decoded_json->{SOME_KEY}, instead of via the dumper. You may have to adjust for encoding layers, but it looks like you’re doing it right so far.

[reply]
[d/l]
[select]
      yeah, you 're right... if don't go with Dumper everything seems ok... i can't start describing how angry i feel for all this time i spent blaming the json... i think i will go and start a farm or something... computers are not for me :( thanks however
[reply]

        I spent hours one evening trying to fix a bug caused by a missing } that happened to fall where the syntax checks refused to tell me accurately where and what was going on (this was an older perl that was weak in this area). Whenever I changed something, the error jumped to a new place. Hours. We all have these moments. Those who are too stubborn to give up start to get paid for it eventually. :P

[reply]
[d/l]
        Same experience just end of last week and beginning of this week: I wasted at least 4 to 4.5 days on a PL/SQL script (a rather simple one, about I would say 300 lines of code) that just stubbornly refused to do what I wanted (I could have made the script in 3 to 4 hours, perhaps even less, in Perl using DBI, but the client did not want that, they insisted on me using PL/SQL, a language that I have used not very much, and it was at least 5 years ago the last time I used it). I stumbled for days on a really stupid problem, namely that if an exception is raised by Oracle (say "no data found"), then the rest of the code of the procedure if not executed, even if you do what is needed to manage the exception. I did not know that, or had forgotten, and did not have a coworker who could help me, we are not using PL/SQL regularly. And it took me ages of debugging time, printing out variables values every other instruction, before I understood what was going wrong. Yes, I was really getting frustrated and angry at not being to solve the problem, but the upside is that I was so happy yesterday when I finally found the solution that I now almost feel good about the whole anecdote. Sorry it that sounds slightly off-topic, my example is not Perl, but I think that you sometimes have these problems and it is good to share about them.
[reply]

Back to Seekers of Perl Wisdom