Yes here is the benchmark that I ran. Fixed string grep is crazy fast. This is run on a 500MB file.

#!/usr/bin/perl

use strict;
use Benchmark qw(cmpthese);

my $fileName = "data2";
my $string = "test";

print "case1 finds ", &case1, " matches \n";
print "case2 finds ", &case2, " matches \n";

cmpthese (10, {
    case0 => sub {&case0},
    case1 => sub {&case1},
    case2 => sub {&case2},
    },
);

sub case0 {
    open(my $file, '<', "$fileName") or die $!;

    while (<$file>) {
    }
}

sub case1 {
    open(my $file, '<', "$fileName") or die $!;
    my $matchCount = 0;

    while (<$file>) {
        $matchCount++ if ($_ =~ /$string/o);
    }

    return $matchCount;
}

sub case2 {
    my $matchCount = `grep -c -F "$string" $fileName`;
    chomp $matchCount;

    return $matchCount;
}



case1 finds 354 matches
case2 finds 354 matches

          s/iter     case1 emptyLoop     case2
case1      8.08        --       -36%      -92%
case0      5.16       57%        --       -88%
case2      0.632     1178%      716%      --
[download]

Hm. On my Win64 system the difference between Perl and grep is much smaller.

[15:00:05.53] c:\test>perl -nle"/234/ && ++$c }{ print $c" 1GB.dat
6791

[15:00:17.31] 
##     11.78 seconds

[15:01:05.67] c:\test>grep -c -F 234 1GB.dat
6791

[15:01:14.28] c:\test>
##      8.61 seconds
[download]

I don't see much wrong with your benchmark, so I wonder if that means your Perl is much slower than mine or my grep slower than yours?

---

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

Yeah I've tried this on two different systems. One a P4 Ubuntu machine (this one), and another an enterprise Redhat server. Both of them showed similar results.

Regardless do you have an idea how to do the array comparison without using a module? I can't install a module either. If I can just find an efficient way to run through the first array and get the two closest numbers from the other array then I can speed test the complete script using grep and your solution. And both of the arrays are already sorted

Okay. If both arrays are sorted, you don't need a binary search.

You just move through them in parallel and you're done in a single pass:

#! perl -slw
use strict;

my @matches = map $_ * 10, 1 .. 10;
my @headers = map $_ * 5 +2, 0 .. 20;

my $hdr = 0;
for my $match ( @matches  ) {
    $hdr++ while $headers[ $hdr + 1 ] < $match;
    printf "match at %d in section from  %d - %d\n",
        $match, $headers[ $hdr ], $headers[ $hdr + 1 ] -1;
}

__END__
[16:20:40.99] c:\test>junk36
match at 10 in section from  7 - 11
match at 20 in section from  17 - 21
match at 30 in section from  27 - 31
match at 40 in section from  37 - 41
match at 50 in section from  47 - 51
match at 60 in section from  57 - 61
match at 70 in section from  67 - 71
match at 80 in section from  77 - 81
match at 90 in section from  87 - 91
match at 100 in section from  97 - 101
[download]

---

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

Yes, even you can use CPAN, also local::lib and even App::perlbrew.

Some thoughts on your benchmark!

Depending on the operating system you're working on, the actual results of your benchmark could be very different. If your benchmark was run on a *nix system, your first call to the subroutine &case1:

print "case1 finds ", &case1, " matches \n";

causes the file to be read and cached by the *nix system. I wasn't sure how perl would or would not benefit from this, but a *nix grep will be able to do pattern matching on the file in cached memory. grep doesn't even have to do a memory to memory copy/move. ( I didn't look at the code, so grep may be doing the memory to memory copy/move. )

I ran you're benchmark on an AIX system, and the results were basically the same as what you saw before. I then modified you're script to call &case2 first, and then &case1 and then &case0 (only once, Benchmark complained!) on a new and un-cached file. The result was that &case0 was the fastest, followed by &case1 and the slowest was &case2(grep). I ran this script on OpenSUSE with similar results. It does appear that perl does get benefit of the caching. If I ran the test again, grep was the winner!

If you're used a *nix system, I hope this gives some idea of why grep looked so much faster than perl.

Note: It's faster to work in memory than on disk.

Further note: You may have to restart the system to guarantee the file isn't cached already. I made this mistake the first time by using a large .gz file that I unzipped, which caused the zipped and unzipped files to be cached.

Thank you

"Well done is better than well said." - Benjamin Franklin

Sure, Unix (and I'd be amazed if there are modern OSses that don't do this) caches files. But it will do so regardless of the program that uses the file. It's not going to say, "Ooooh, this file is opened by a process called 'grep', I better cache the results, and here, this file is opened by dirty sticking little perl, I'm not going to keep that one around!".

What I am trying to do is leverage the speed of grep to complete the task. The data files are huge and I have found that grep is significantly faster than even running the files through an empty Perl loop.

Is grep twice as fast as perl? Because if it isn't making two passes and then a binary search two match the lines numbers, and then a partial third pass to extract the required lines doesn't make sense.