IB2017 has asked for the wisdom of the Perl Monks concerning the following question:

I am a bit confused by the error I get when using my $decoded_json = decode_json( $json_text );. I retrieve $json_text with:

EDIT

use strict;
use warnings;
use LWP::UserAgent;
use Data::Dumper;
use JSON::PP;
use HTTP::Request::Common;

my $query = "research";
my $format = "";
my $domain = "";
my $language = "";
    
my $data = { query => $query, format => $format, domain => $domain, la
+nguage => $language };

my $url = 'MyDomain/cgi-bin/querySearchEngine.pl';
my $ua  = LWP::UserAgent->new;
my $request = POST($url, Content_Type=>'application/json',
    Content => encode_json($data) );
my $response = $ua->request($request);
    
if ($response->is_success) {
    my $json_text= $response->content;    
    my $decoded_json = decode_json( $json_text );
}

[download]

My response $json_text (an array with a list of URI) looks like this:

{"myData":["https://www.sante.de/fileadmin/user_upload/Stellenanzeige_
+Mitarbeiter_Research__Development.pdf","https://www.national-bank.de/
+fileadmin/user_upload/nationalbank/Vermoegensmanagement/Research/Kapi
+talmarkt_Studien/Kapitalmarktmonitor_Dezember_2018.pdf","https://www.
+acxit.com/en/wp-content/uploads/2018/12/CAREERS_Stellenanzeige_WS_Res
+earch_Deal_Generation_20181211.pdf","https://www.s-a.uni-muenchen.de/
+studierende/jobboerse/jobangebote/aktuelle/20181221_44.pdf","https://
+www.dbresearch.de/PROD/RPS_DE-PROD/PROD0000000000459596/Deutscher_H%C
+3%A4user-_und_Wohnungsmarkt_2018.pdf","https://www.bayernlb.de/intern
+et/media/de/ir/downloads_1/bayernlb_research/multiasset_produkte/Pers
+pektiven.pdf","https://www.borders-in-motion.de/documents/12991/34723
+4/Flyer_Research_Factory_WS18-19.pdf/2baa6987-219a-4cbb-8ab0-da36754f
+1282","https://deka.de/site/dekade_privatkunden_site/get/documents/de
+kade/Publikationen/2018/Volkswirtschaft/Aktuell/20181220_VA_USA_Zinse
+ntscheid.pdf","https://www.dmtm.com/medienverwaltung/medien/181218_Re
+search_Engineer_Website.pdf","https://www.ewi.research-scenarios.de/c
+ms/wp-content/uploads/2018/08/Czock_CV_de_20181214.pdf"]}

[download]

Server side EDIT

#!/usr/bin/perl
use strict;
use warnings;
use CGI qw(:standard);
use JSON;
use utf8;
use LWP::UserAgent;
use HTTP::Request::Common;
use Data::Dumper;

my $q = CGI->new();
print $q->header;
my $data = $q->param('POSTDATA');
$data = decode_json($data) if $data;

my $query = $data->{'query'};
my $format = $data->{'format'};
my $domain = $data->{'domain'};
my $language = $data->{'language'};

my @UriCollected;

print "Content-type: application/json;\n\n";

#performing crawling here and collecting uri in @UriCollected
print objToJson( { myData => \@UriCollected } );

[download]

Error is:

malformed JSON string, neither array, object, number, string or atom, 
+at character offset 0 (before "Content-type: applic...") at

[download]

Error line my $decoded_json = decode_json( $json_text );

I use a similar way to exchange information in many cgi scripts with no problems. So I must be doing a very stupid error here... but I do not understand where...

Replies are listed 'Best First'.
Re: json Array with URI
by haukex (Archbishop) on Dec 27, 2018 at 15:01 UTC

    The message you're getting (at character offset 0 (before "Content-type: applic...")) makes it sound like there's something wrong with $response->content, but you haven't shown what module you're using to decode the JSON, what $json_text actually contains (using e.g. Data::Dumper or Data::Dump), or what URL you're fetching it from. What you have shown works fine for me. Please provide a runnable SSCCE that we can use to reproduce the error.

    use warnings;
use strict;
use Data::Dump;
use JSON::MaybeXS qw/decode_json/;

my $json_text = do { local $/; <DATA> };
dd decode_json($json_text);

__DATA__
{"myData":["https://www.sante.de/fileadmin/user_upload/Stellenanzeige_
+Mitarbeiter_Research__Development.pdf","https://www.national-bank.de/
+fileadmin/user_upload/nationalbank/Vermoegensmanagement/Research/Kapi
+talmarkt_Studien/Kapitalmarktmonitor_Dezember_2018.pdf","https://www.
+acxit.com/en/wp-content/uploads/2018/12/CAREERS_Stellenanzeige_WS_Res
+earch_Deal_Generation_20181211.pdf","https://www.s-a.uni-muenchen.de/
+studierende/jobboerse/jobangebote/aktuelle/20181221_44.pdf","https://
+www.dbresearch.de/PROD/RPS_DE-PROD/PROD0000000000459596/Deutscher_H%C
+3%A4user-_und_Wohnungsmarkt_2018.pdf","https://www.bayernlb.de/intern
+et/media/de/ir/downloads_1/bayernlb_research/multiasset_produkte/Pers
+pektiven.pdf","https://www.borders-in-motion.de/documents/12991/34723
+4/Flyer_Research_Factory_WS18-19.pdf/2baa6987-219a-4cbb-8ab0-da36754f
+1282","https://deka.de/site/dekade_privatkunden_site/get/documents/de
+kade/Publikationen/2018/Volkswirtschaft/Aktuell/20181220_VA_USA_Zinse
+ntscheid.pdf","https://www.dmtm.com/medienverwaltung/medien/181218_Re
+search_Engineer_Website.pdf","https://www.ewi.research-scenarios.de/c
+ms/wp-content/uploads/2018/08/Czock_CV_de_20181214.pdf"]}

    [download]

    Update: I see Corion made many of the same points.

[reply]
[d/l]
[select]
Re: json Array with URI
by Corion (Patriarch) on Dec 27, 2018 at 14:59 UTC

    Maybe the error comes from:

    Content => encode_json($data)

    [download]

    You don't show us what is in $data and you don't show us the line number where the error occurs. The error message (before "Content-type: applic...") suggests that you pass a string containing HTTP headers to decode_json somewhere, but it's hard to know what you're doing wrong when you are not showing us the relevant code, data and error message.

    If decode_json( $json_text ); fails, then you have something in $json_text that is not valid JSON. Inspecting $json_text will tell you more about it.

[reply]
[d/l]
[select]
Re: json Array with URI
by IB2017 (Pilgrim) on Dec 27, 2018 at 15:10 UTC

    Thank you for pointing me to incomplete code. I updated my post with some more code (stripping it down from my code base). However, I posted the $response->content; held in variable $json_text. (I thought this is the most important part to understand why decode_json fires an error. But from your answers, I start to think the problem must be some where else. (disclaimer: I am not a json expert, just starting using it - quite successfully - a couple of weeks ago)

[reply]
[d/l]
[select]

      In your server code you have the following line:

      print "Content-type: application/json;\n\n";

      [download]

      What is that line doing there and are you certain that the rest of the script hasn't already sent complete headers? This would easily explain why $response->decoded_content starts with Content-type:.

      Update: And indeed, in your server code, you have print $q->header;. Printing more headers after that will not add those lines to the headers but to the reply body. See the ->content_type method of CGI.

      This also makes your diagnosis of

      My response $json_text (an array with a list of URI) looks like this:
      somewhat suspect because there must (or rather, should) be a line Content-type: application/json; right before that.

[reply]
[d/l]
[select]

        Now I see. You are right. I was printing twice the header. I know it was something stupid somewhere. Thank you very much for opening my eyes.

[reply]
      However, I posted the $response->content; held in variable $json_text. (I thought this is the most important part to understand why decode_json fires an error.

      As Corion and I already hinted at, it's pretty unlikely that you posted exactly what $json_text contains. Please use a module such as Data::Dumper or Data::Dump.

      To output a CGI header with a different Content-Type, you can say print $q->header("application/json");

[reply]
[d/l]
[select]

        Thank you for this hint. I posted what a simple print gave me. print Dumper is of course more complete. Learnt a lot, as always.

[reply]
[d/l]
[select]

Back to Seekers of Perl Wisdom