ron800 has asked for the wisdom of the Perl Monks concerning the following question:

Hello, I've searched through this group, but no one seems to have the same issue. I want to parse large JSON files to see if the syntax is correct. I found examples but they all seem to want to parse from a string. I want to do something like this: 
use JSON::Parse 'assert_valid_json';
 unless (valid_json ($json)) {
        # do something }
    eval {
        assert_valid_json ('["xyz":"b"]');
    };
    if ($@) {
        print "Your JSON was invalid: $@\n";
    }
  }

[download]
My files are multiple megabytes and reading them into a string for parsing is not realistic. Any suggestions would be appreciated.

Replies are listed 'Best First'.
Re: Verify syntax of large JSON files
by Loops (Curate) on Nov 23, 2014 at 04:40 UTC

    You could use one of the JSON readers that have a streaming capability rather than loading the entire thing into memory, such as JSON::Streaming::Reader. The code below is a quick try that works here, although I don't have any multi-meg JSON files handy. Have to provide routines for each element type read in the stream; the code below just has null subs for them but doing a $jsonr->skip instead would probably speed things up even more:

    use IO::File;
use JSON::Streaming::Reader;

sub is_valid_json {
    my $fh = IO::File->new(shift, "r");
    my $jsonr = JSON::Streaming::Reader->for_stream($fh);
    my @ignore = map {($_, sub {})}
        qw(start_array end_array start_object add_string add_number
           add_boolean add_null end_object start_property end_property
+);
    eval {$jsonr->process_tokens(@ignore, error => sub {die "@_"})};
    return !$@;
}

print is_valid_json('file.json') ? 'ok' : 'panic';

    [download]
[reply]
[d/l]
      Thanks. That seems to work on my small files. The only issue is that I can't figure out how to indicate where exactly in the JSON file the error occurs.
[reply]

        Looking at JSON::Streaming::Reader, I quickly found this line: 

            my $char = $self->_peek_char();

        [download]

        which lead me to expect the module to be quite slow at what it does.

        Luckily, parsing JSON is quite an easy task. Just checking it for validity is even easier than that. So I quickly rolled a JSON parser that just checked validity and didn't read the whole file into RAM at once.

        So I built a 65MB file of JSON so I could compare: 

         > perl isJson.pl < file.json
Valid JSON.
Took 38s.

 > perl loops.pl
okTook 446s.

        [download]

        So it looks like my version is over 10x faster.

        [Update: Adding $jsonr->skip() inside the empty 'sub' as Loops suggests, makes the module run about 2x faster: 

         > perl skip.pl
okTook 235s.

        [download]

        or only about 1/6th as fast as my code.]

        Then I added an error near the end of the file: 

         > perl loops.pl
panicTook 402s.

 > perl isJson.pl < file.json
Invalid bare word (troo) at line 1020000, pos 8 (t)
Took 39s.

        [download]

        So you can see it also gives much better information about where the problem was.

        Read more... (3 kB)

        - tye        

[reply]
[d/l]
[select]

Back to Seekers of Perl Wisdom