ve6sar has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks, I've spent several hours over the last couple days trying to get this to work. My script connects to a website that streams JSON continuously, what I need to do is decode the JSON, filter it then insert the filtered results into a database. The script below currently connects but JSON::SL doesn't seem to be identifing the JSON objects and decoding them the script just sits there waiting to decode.
#!/usr/bin/perl use strict; use IO::Socket::SSL; #qw(debug3); use JSON::SL; print "JSON Stream parser test\r\n"; my $p = JSON::SL->new; #Look for everything past the first JSON pointer $p->set_jsonpointer(["/^"]); # simple client my $cl = IO::Socket::SSL->new( PeerAddr => 'example.com', PeerService => 'https', PeerPort => 443, Proto => 'tcp', Reuse => 1 ); die "no connection" unless $cl -> connected(); print $cl "GET /stream/report?token=IAMAFAKETOKEN HTTP/1.0\r\n\r\n +"; print <$cl>; local $/ = \5; #read only 5 bytes at a time while (my $line = <$cl>) { $p->feed($line); #parse what you can fetch anything that comple +ted the parse and matches the JSON Pointer while (my $obj = $p->fetch) { print "$obj->{Value}{sequenceNumber}: $obj->{Value}{sender +Callsign}\n"; #Inserted for testing #do filtering and database stuff here } } close $cl;
Using this code to test strictly the decode it works.
#!/usr/bin/perl use strict; use warnings; use JSON::SL; my $p = JSON::SL->new; #look for everthing past the first level (i.e. everything in the array +) $p->set_jsonpointer(["/^"]); local $/ = \5; #read only 5 bytes at a time while (my $buf = <DATA>) { $p->feed($buf); #parse what you can #fetch anything that completed the parse and matches the JSON Poin +ter while (my $obj = $p->fetch) { print "$obj->{Value}{sequenceNumber}: $obj->{Value}{senderCall +sign}\n"; } } __DATA__ [ {"sequenceNumber":1401615035,"frequency":7076402,"mode":"JT65","sNR +":-1,"flowStartSeconds":1492192912,"senderCallsign":"DJ0FX","senderLo +cator":"JN67KT","receiverCallsign":"OZ7IO","receiverLocator":"JO65gq" +,"receiverDecoderSoftware":"WSJT-X v1.7.0 r7405"}, {"sequenceNumber":1401615039,"frequency":7077903,"mode":"JT65","sN +R":-11,"flowStartSeconds":1492192973,"senderCallsign":"R7NO","senderL +ocator":"KN98","receiverCallsign":"OZ7IO","receiverLocator":"JO65gq", +"receiverDecoderSoftware":"WSJT-X v1.7.0 r7405"}, {"sequenceNumber":1401615040,"frequency":7076811,"mode":"JT65","sN +R":-11,"flowStartSeconds":1492193032,"senderCallsign":"F6AVP","sender +Locator":"JN28UX","receiverCallsign":"OZ7IO","receiverLocator":"JO65g +q","receiverDecoderSoftware":"WSJT-X v1.7.0 r7405"} ]
Here's a sample of the data as received, in the working decode script I had to add a comma at the end of each line.
{"sequenceNumber":1401615035,"frequency":7076402,"mode":"JT65","sNR":- +1,"flowStartSeconds":1492192912,"senderCallsign":"DJ0FX","senderLocat +or":"JN67KT","receiverCallsign":"OZ7IO","receiverLocator":"JO65gq","r +eceiverDecoderSoftware":"WSJT-X v1.7.0 r7405"} {"sequenceNumber":1401615036,"frequency":7076996,"mode":"JT65","sNR":- +6,"flowStartSeconds":1492192912,"senderCallsign":"RU6MO","senderLocat +or":"KN97KF","receiverCallsign":"OZ7IO","receiverLocator":"JO65gq","r +eceiverDecoderSoftware":"WSJT-X v1.7.0 r7405"} {"sequenceNumber":1401615037,"frequency":7077360,"mode":"JT65","sNR":- +5,"flowStartSeconds":1492192913,"senderCallsign":"RV3QD","senderLocat +or":"KO91NQ","receiverCallsign":"OZ7IO","receiverLocator":"JO65gq","r +eceiverDecoderSoftware":"WSJT-X v1.7.0 r7405"}

Replies are listed 'Best First'.
Re: Trying to decode JSON streaming from a website
by Corion (Patriarch) on Apr 14, 2017 at 21:31 UTC

    I don't think that JSON::SL likes invalid JSON, and if your stream is coming in without the commas separating the JSON objects, it is not valid JSON.

    You will have to find a way to insert the commas before feeding the data to JSON::SL. Maybe s/\{"sequenceNumber/,\{"sequenceNumber/g is already enough. If you are lucky, the JSON stream really has the newlines as you showed, then you can do s!\n!,!g.

      I added commas to the end of each line, and now JSON::SL is trying to decode at least now I'm getting a hash ref error...
      JSON Stream parser test Received: {"sequenceNumber":1402012831,"frequency":7077409,"mode":"JT65","sNR":- +21,"flowStartSeconds":1492208570,"senderCallsign":"EB5JRL","senderLoc +ator":"IM99SL","receiverCallsign":"ON3HPI","receiverLocator":"JO20et" +,"receiverDecoderSoftware":"JTDX v17.7"}, Can't use string ("1402012831") as a HASH ref while "strict refs" in u +se at client.pl line 38.
      Should it not be using "sequenceNumber" as the reference and "1402012831" as the data? The line it fails on is  print "$obj->{Value}{flowStartSeconds}: $obj->{Value}{senderCallsign}\n"; If I use Data::Dumper the first element looks like below, could it be I've got my JSON pointer wrong so it's not looking at the whole line?
      $VAR1 = { 'Value' => 1402024539, 'Path' => '/sequenceNumber', 'JSONPointer' => '/^' };
      I'm not overly strong in OO Perl, I know I should how to use it more. Thanks

        I don't know JSON::SL, but it seems that you have to configure it better to call your code at different times than it currently does.

        I don't know what kind of JSON path spec it implements, but have you tried just using / instead of /^?

        If you post a short, self-contained program that still reproduces the error, then we can give help more to the point than just guess about your data, your code and the behaviour of JSON::SL. See also SSCCE.

Re: Trying to decode JSON streaming from a website
by huck (Prior) on Apr 14, 2017 at 21:41 UTC

    Besides what Corion said Re: Trying to decode JSON streaming from a website, your __data__stream starts with a [ and that goes with

    #Look for everything past the first JSON pointer $p->set_jsonpointer(["/^"]);
    But your socket stream doesnt seem too. Possibly
    $p->feed('['); while (my $line = <$cl>) { ...
    would help

    or MAYBE if you drop $p->set_jsonpointer(["/^"]); you wont need to prefeed it or the inserted commas, dunnno

Re: Trying to decode JSON streaming from a website
by Anonymous Monk on Apr 14, 2017 at 21:29 UTC
    Are you actually receiving any data? Maybe put a print in there...
    while (my $line = <$cl>) { print "Received: $line\n"; ... }
      I removed the line "print <$cl>;" and now I'm receiving data so its a step in the right direction. Thanks
        Thanks for every ones help. I got it working here's the working code. I had to remove from the original code the line with print <$cl>; I also had to alter each line to have a leading [ and a trailing ], it's now working. Working code:
        #!/usr/bin/perl use strict; use IO::Socket::SSL; #qw(debug3); use JSON::SL; use Data::Dumper; print "JSON Stream parser test\r\n"; my $p = JSON::SL->new; #Look for everything past the first JSON pointer $p->set_jsonpointer(["/^"]); # simple client my $cl = IO::Socket::SSL->new( PeerAddr => 'example.com', PeerService => 'https', PeerPort => 443, Proto => 'tcp', Reuse => 1 ); die "no connection" unless $cl -> connected(); print $cl "GET /stream/report?token=IMNOTAREALTOKEN HTTP/1.0\r\n\r +\n"; #print <$cl>; #local $/ = \5; #read only 5 bytes at a time while (my $line = <$cl>) { chomp($line); #Remove new line characters $line = "[$line],\n"; # Add leading [ and trailing ], so i +t will parse correctly if ($line =~ m/{"\w{14}/) { print "Received:\n$line"; $p->feed($line); #parse what you can fetch anything that co +mpleted the parse and matches the JSON Pointer while (my $obj = $p->fetch) { print Dumper($obj); print "$obj->{Value}{flowStartSeconds}: $obj->{Value}{ +senderCallsign}\n"; } } } close $cl;