Hm. On my Win64 system the difference between Perl and grep is much smaller.

[15:00:05.53] c:\test>perl -nle"/234/ && ++$c }{ print $c" 1GB.dat
6791

[15:00:17.31] 
##     11.78 seconds

[15:01:05.67] c:\test>grep -c -F 234 1GB.dat
6791

[15:01:14.28] c:\test>
##      8.61 seconds
[download]

I don't see much wrong with your benchmark, so I wonder if that means your Perl is much slower than mine or my grep slower than yours?

Regardless do you have an idea how to do the array comparison without using a module? I can't install a module either. If I can just find an efficient way to run through the first array and get the two closest numbers from the other array then I can speed test the complete script using grep and your solution. And both of the arrays are already sorted

Okay. If both arrays are sorted, you don't need a binary search.

You just move through them in parallel and you're done in a single pass:

#! perl -slw
use strict;

my @matches = map $_ * 10, 1 .. 10;
my @headers = map $_ * 5 +2, 0 .. 20;

my $hdr = 0;
for my $match ( @matches  ) {
    $hdr++ while $headers[ $hdr + 1 ] < $match;
    printf "match at %d in section from  %d - %d\n",
        $match, $headers[ $hdr ], $headers[ $hdr + 1 ] -1;
}

__END__
[16:20:40.99] c:\test>junk36
match at 10 in section from  7 - 11
match at 20 in section from  17 - 21
match at 30 in section from  27 - 31
match at 40 in section from  37 - 41
match at 50 in section from  47 - 51
match at 60 in section from  57 - 61
match at 70 in section from  67 - 71
match at 80 in section from  77 - 81
match at 90 in section from  87 - 91
match at 100 in section from  97 - 101
[download]

---

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

Excellent! That looks way more efficient. I will let you know how it works out.

Well Browser, it seems that you were right and my line tracking idea didn't work out so well. Even with your more efficient code, it just took forever. Grep was very fast, but the line tracking combined with outputting certain lines of the file was extremely slow. What I ended up doing was a combination of grep (using the 50 line context option), and a foreach loop on the resulting array data. I parsed the data that way and it's very fast. On that slow P4/512MB machine I can parse a 1GB file for a fixed string in about 15 seconds. The same parsing done completely with a Perl file loop takes takes 15x as long.

I can only speculate but Perl needs to read the entire file line by line and search for the string, while grep obviously does something far more efficient to find the string. In the end, I haven't yet found anything close to the speed of grep to parse large 1GB files.

If anyone out there can find a faster way, I would be impressed. Perhaps there is a Perl module that can do very fast searches on files? AWK? I think that's comparable in terms of speed to grep, and I could also parse the data with it... hmmm..

Yes, even you can use CPAN, also local::lib and even App::perlbrew.

Some thoughts on your benchmark!

Depending on the operating system you're working on, the actual results of your benchmark could be very different. If your benchmark was run on a *nix system, your first call to the subroutine &case1:

causes the file to be read and cached by the *nix system. I wasn't sure how perl would or would not benefit from this, but a *nix grep will be able to do pattern matching on the file in cached memory. grep doesn't even have to do a memory to memory copy/move. ( I didn't look at the code, so grep may be doing the memory to memory copy/move. )

I ran you're benchmark on an AIX system, and the results were basically the same as what you saw before. I then modified you're script to call &case2 first, and then &case1 and then &case0 (only once, Benchmark complained!) on a new and un-cached file. The result was that &case0 was the fastest, followed by &case1 and the slowest was &case2(grep). I ran this script on OpenSUSE with similar results. It does appear that perl does get benefit of the caching. If I ran the test again, grep was the winner!

If you're used a *nix system, I hope this gives some idea of why grep looked so much faster than perl.

Note: It's faster to work in memory than on disk.

Further note: You may have to restart the system to guarantee the file isn't cached already. I made this mistake the first time by using a large .gz file that I unzipped, which caused the zipped and unzipped files to be cached.

Thank you

Sure, Unix (and I'd be amazed if there are modern OSses that don't do this) caches files. But it will do so regardless of the program that uses the file. It's not going to say, "Ooooh, this file is opened by a process called 'grep', I better cache the results, and here, this file is opened by dirty sticking little perl, I'm not going to keep that one around!".

You may want to consider the different types of disk I/O sub-systems in modern operating systems. Most *nix systems have Raw I/O support, Direct I/O support, Concurrent I/O support, Modular I/O support, etc. For example, most databases use raw I/O. Performance of these applications is usually better when using raw I/O rather than using other I/O methods, because it avoids the additional work of memory copies, logging, and inode locks.

My comment about perl was directed at which I/O subsystem perl was using, not that perl would be treated differently by that I/O sub-system.

When writing *nix utilities ( like grep ), system programmers were encouraged to write "cache aware" programs. Whenever possible, work on the cached version directly, and avoid memory to memory move/copy. ( For clarification, I use "move/copy" because the operating system may perform a move rather than a copy, but this happens in the paging I/O sub-system, and has to do with paging performance. This is transparent to the application. )

All I was trying to point out, was that a 500MB file on a test machine may be cached, but may not be cached on a production machine, and that a pure perl solution may very well be a better solution on a production machine. But that is the decision of the OP

"Well done is better than well said." - Benjamin Franklin