in reply to Re: Verify syntax of large JSON files
in thread Verify syntax of large JSON files

Thanks. That seems to work on my small files. The only issue is that I can't figure out how to indicate where exactly in the JSON file the error occurs.
  • Comment on Re^2: Verify syntax of large JSON files

Replies are listed 'Best First'.
Re^3: Verify syntax of large JSON files (code)
by tye (Sage) on Nov 23, 2014 at 22:58 UTC

    Looking at JSON::Streaming::Reader, I quickly found this line:

    my $char = $self->_peek_char();

    which lead me to expect the module to be quite slow at what it does.

    Luckily, parsing JSON is quite an easy task. Just checking it for validity is even easier than that. So I quickly rolled a JSON parser that just checked validity and didn't read the whole file into RAM at once.

    So I built a 65MB file of JSON so I could compare:

    > perl isJson.pl < file.json Valid JSON. Took 38s. > perl loops.pl okTook 446s.

    So it looks like my version is over 10x faster.

    [Update: Adding $jsonr->skip() inside the empty 'sub' as Loops suggests, makes the module run about 2x faster:

    > perl skip.pl okTook 235s.

    or only about 1/6th as fast as my code.]

    Then I added an error near the end of the file:

    > perl loops.pl panicTook 402s. > perl isJson.pl < file.json Invalid bare word (troo) at line 1020000, pos 8 (t) Took 39s.

    So you can see it also gives much better information about where the problem was.

    - tye        

      I decided I had to front-page this thread, just so more people would see your json parser. Thank you for posting this.