in reply to Perl Script performance issue

Hello Tara, and welcome to the Monastery!

One obvious slow-down is the shell-out call to system grep:

$final_data = ` grep "$key_pattern" "${indir}/${lkp_file_name}" |cut - +d"|" -f"$data_location"`;

which is expensive in itself and rendered especially so by being nested inside three loops. Try replacing it with a call to Perl’s inbuilt grep and see if that produces a significant speedup.

Hope that helps,

Athanasius <°(((><contra mundum Iustus alius egestas vitae, eros Piratica,

Replies are listed 'Best First'.
Re^2: Perl Script performance issue
by Tara (Initiate) on Dec 17, 2015 at 12:39 UTC

    Perl's in built grep works on elements. How do i make i behave the way i intend? Do i have to store the line into an array or something? Confused..please help..

      Hello again Tara,

      You have the line:

      $final_data = ` grep "$key_pattern" "${indir}/${lkp_file_name}" |cut - +d"|" -f"$data_location"`;

      Here is one way to write this in pure Perl (untested):

      my $file = $indir . '/' . $lkp_file_name; open(my $fh, '<', $file) or die "Cannot open file '$file' for reading: $!"; my @matched_lines = grep { /\Q$key_pattern/ } <$fh>; close $fh or die "Cannot close file '$file': $!"; my @fields; for my $line (@matched_lines) { my $field = (split /\|/, $line)[$data_location - 1]; chomp $field; push @fields, $field; } $final_data = join "\n", @fields;

      (There should also be some error handling to deal with the possibility that a matched line contains an insufficient number of fields.)

      Hope that helps,

      Athanasius <°(((><contra mundum Iustus alius egestas vitae, eros Piratica,