Re: slow parser how to make it faster

First, split takes a regular expression, not a string (except sometimes ;-). So splitting on '\t' won't give the expected results. Next, I suspect that @id_list grows fairly long over time, and you are iterating over the entire array with grep. A hash is more suitable here:

use strict;
use warnings;


my $name = '';
my %id_list;

open (STDIN,"perl first_program.pl|") || die "$!";
open (OUT, ">", "out.txt") || die "$!";

while(my $line_in =<STDIN>){
  chomp($line_in);
  my @line_array = split(/\t/,$line_in);
  my @subline_array = split(/\|/, $line_array[1]);

  unless ($name eq $line_array[0]) {
    %id_list = ();
    $name = $line_array[0];  
  }
  next if $id_list{$subline_array[3]}++;
  print OUT "$line_in\n";
}
[download]

If you experience memory shortage, tie the hash %id_list to a disk file via e.g. DB_File.

Comment on Re: slow parser how to make it faster Select or Download Code

Replies are listed 'Best First'.
Re^2: slow parser how to make it faster by ig (Vicar) on Jun 11, 2009 at 08:39 UTC
And perhaps a little faster with an RE instead of splits. `use strict; use warnings; my $name = ''; my %id_list; open (STDIN,"perl first_program.pl\|") \|\| die "$!"; open (OUT, ">", "out.txt") \|\| die "$!"; while( my $line_in = <STDIN> ) { line_in =~ m/^([^\t])(?:[^\\|]\\|){3}([^\\|])/; unless($name eq $1) { %id_list = (); $name = $1; } next if $id_list{$2}++; print OUT "$line_in"; }` [download] Although the parsing of the line is quicker with the RE, the difference may be very small compared to the I/O time. The benefit will depend very much on how often the loop prints. `use strict; use warnings; use Benchmark; my $line_in = "mmenr\thh\|gg\|kk\|3445\|uu\|zzz\t234\twwe\twe\tqw\t233\n"; Benchmark::cmpthese( 1000000, { 'split' => sub { chomp($line_in); my @line_array = split(/\t/,$line_in); my @subline_array = split(/\\|/, $line_array[1]); }, 're' => sub { my ($name, $id) = ($line_in =~ m/^([^\t])(?:[^\\|]\\|){3}([^\\|] +)/); } }); __END__ Rate split re split 84962/s -- -68% re 266667/s 214% --` [download]	[reply] [d/l] [select]

Replies are listed 'Best First'.

Re^2: slow parser how to make it faster
by ig (Vicar) on Jun 11, 2009 at 08:39 UTC

And perhaps a little faster with an RE instead of splits.

use strict;
use warnings;

my $name = '';
my %id_list;

open (STDIN,"perl first_program.pl|") || die "$!";
open (OUT, ">", "out.txt") || die "$!";

while( my $line_in = <STDIN> ) {
    line_in =~ m/^([^\t]*)(?:[^\|]*\|){3}([^\|]*)/;
    unless($name eq $1) {
        %id_list = ();
        $name = $1;
    }
    next if $id_list{$2}++;
    print OUT "$line_in";
}
[download]

Although the parsing of the line is quicker with the RE, the difference may be very small compared to the I/O time. The benefit will depend very much on how often the loop prints.

use strict;
use warnings;
use Benchmark;

my $line_in = "mmenr\thh|gg|kk|3445|uu|zzz\t234\twwe\twe\tqw\t233\n";

Benchmark::cmpthese( 1000000, {
    'split' => sub {
      chomp($line_in);
      my @line_array = split(/\t/,$line_in);
      my @subline_array = split(/\|/, $line_array[1]);
    },
    're' => sub {
      my ($name, $id) = ($line_in =~ m/^([^\t]*)(?:[^\|]*\|){3}([^\|]*
+)/);
    }
});
__END__
          Rate split    re
split  84962/s    --  -68%
re    266667/s  214%    --
[download]

[reply]
[d/l]
[select]