Looking at JSON::Streaming::Reader, I quickly found this line: 

    my $char = $self->_peek_char();

[download]

which lead me to expect the module to be quite slow at what it does.

Luckily, parsing JSON is quite an easy task. Just checking it for validity is even easier than that. So I quickly rolled a JSON parser that just checked validity and didn't read the whole file into RAM at once.

So I built a 65MB file of JSON so I could compare: 

 > perl isJson.pl < file.json
Valid JSON.
Took 38s.

 > perl loops.pl
okTook 446s.

[download]

So it looks like my version is over 10x faster.

[Update: Adding $jsonr->skip() inside the empty 'sub' as Loops suggests, makes the module run about 2x faster: 

 > perl skip.pl
okTook 235s.

[download]

or only about 1/6th as fast as my code.]

Then I added an error near the end of the file: 

 > perl loops.pl
panicTook 402s.

 > perl isJson.pl < file.json
Invalid bare word (troo) at line 1020000, pos 8 (t)
Took 39s.

[download]

So you can see it also gives much better information about where the problem was.

Here is the code: 

#!/usr/bin/perl -w
use strict;

END {
    warn "Took ", time()-$^T, "s.\n";
}

sub fail {
    my( $off, @msg ) = @_;
    my $line = $.;
    die @msg, " at line $line (end of file)\n"
        if  ! defined $off;
    my $pos = pos $_ || 0;
    my $char = substr( $_, $pos+$off, 1 );
    $pos++;
    die @msg, " at line $line, pos $pos ($char)\n";
}

my $v1 = @ARGV ? 1 : 0;
my %bare = qw< null 1 true 1 false 1 >;
my $num = qr{
    -?
    (?:
        0
     |  [1-9][0-9]*
    )
    (?: [.][0-9]+ )?
    (?: [eE][-+]?[0-9]+ )?
}x;
my @stack;
my $end = '1';
while( <STDIN> ) {
    /\G\s+/gc;
    while( ! /\G$/gc ) {
        my $in = $stack[-1] || '';
        if( '"' eq $end ) {
            while( /\G(?:[^\\"]+|\\(.))/gcs ) {
                ;
            }
            fail( -1, "Trailing \\ at end of file" )
                if  /\G\\$/gc;
            $end = ','
                if  /\G"/gc;
        } elsif( ':' eq $end ) {
            fail( 0, "Expected colon" )
                if  ! /\G:/gc;
            push @stack, ':';
            $end = '';
        } elsif( /\G([\]\}])/gc ) {
            my $got = $1;
            my $want = $end !~ /[,2]/ || ':' eq $in
                ? 'value' : $in || 'end of file';
            fail( -1, "Found close bracket, $got, when expecting $want
+" )
                if  $got ne $want;
            pop @stack;
            $end = ',';
        } elsif( ',' eq $end ) {
            fail( 0, "Expected end of file" )
                if  ! $in;
            fail( 0, "Expected comma" )
                if  ! /\G,/gc;
            $end = '';
        } elsif( "\}" eq $in ) {
            fail( 0, "Expected string (key)" )
                if  ! /\G"/gc;
            $end = '"';
        } elsif( /\G([\[\{])/gc ) {
            push @stack, {qw< [ ] { } >}->{$1};
            $end = '2';
        } elsif( '1' eq $end && $v1 ) {
            fail( 0, "JSON of just a scalar not allowed" );
        } elsif( /\G"/gc ) {
            $end = '"';
        } elsif( /\G$num/gc ) {
            $end = ',';
        } elsif( /\G([a-z]+)/gc ) {
            fail( -length($1), "Invalid bare word ($1)" )
                if  ! $bare{$1};
            $end = ',';
        } else {
            fail( 0, "Expected value" );
        }
        if( ',' eq $end && @stack ) {
            $end = ":"
                if  "\}" eq $stack[-1];
            pop @stack
                if  ':' eq $stack[-1];
        }
        /\G\s+/gc;
    }
}
fail( undef, "Empty JSON string" )
    if  1 eq $end;
fail( undef, "Unclosed string" )
    if  '"' eq $end;
fail( undef, "Unclosed aggregate $stack[-1]" )
    if  @stack;
warn "Valid JSON.\n";

[download]

- tye        

In reply to Re^3: Verify syntax of large JSON files (code) by tye
in thread Verify syntax of large JSON files by ron800

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.