parse a 1GB file for a fixed string in about 15 seconds. The same parsing done completely with a Perl file loop takes takes 15x as long.

That suggests that Perl is taking 3 1/2 minutes to read through a 1GB file, which is a nonsense. It takes just 12 seconds on my machine which is nothing special in the disk department:

[ 9:27:30.43] c:\test>perl -nle1 1GB.dat

[ 9:27:42.87] c:\test>
[download]

I can only speculate but Perl needs to read the entire file line by line and search for the string, while grep obviously does something far more efficient to find the string. In the end, I haven't yet found anything close to the speed of grep to parse large 1GB files.

grep cannot avoid reading every line of the file, so the difference is not due to some special grep magic.

A much more likely cause is the way you've coded your perl script.

The simple script I posted above produces all the sections containing a particular search term from a >1GB file in 50 seconds without using anything fancy:

#! perl -sw
use strict;

my $term = shift;

my @section;
while( <> ) {
    @section = () if /Header/;
    push @section, $_;
    if( /$term/ ) {
         print for @section;
         print while defined( $_ = <> ) and not /^Header/;
    }
}
__END__
07/02/2011  09:33     1,090,025,317 886391.dat

[ 9:47:22.80] c:\test>junk34 lima 886391.dat >junk.dat

[ 9:48:12.67] c:\test>
[download]

Here is the code I used just now to test them against each other. The additional operations which I added could certainly affect the benchmark but they are the same in both cases.

use strict;
use warnings;
use Benchmark qw(cmpthese);

my $fileName = "data1";
my $string = $ARGV[0] || die "\nNo Search String Provided.$!\n\n";
my $stringRegex = qr{$string};
my $headerRegex = qw{^Header};  ## Match Headers
my $greenRegex = qr{(^Header).*)};
my $redRegex = qr{($string)};

cmpthese (1, {
    case1 => sub {&case1},
    case2 => sub {&case2},
    },
);

sub case1 {
    my $matches = `grep -h -F -C50 '$grepString' $fileName`;
    my $match; my $data; my $header;

    for (split /^/, $matches) {  ## More efficient than array? (Scalar
+s use less memory)
        if ($_ =~ $headerRegex) {  ## Match headers
            if ($match) {
                $data =~ s/$greenRegex/\033[1;32m$1\033[m/o;  ## Color
+ headers in GREEN
                $data =~ s/$redRegex/\033[1;31m$1\033[m/go;  ## Color 
+string matches in RED
                $data =~ s/\n\n/\n/go;  ## Remove double blanks lines
                print $data, "\n";
                $match = 0;
            }
            $data = ""; $header = 1;
        }
        if (($header) && (0 > index ($_, $string))) {  ## Match body o
+f section
            $data .= $_;
        } elsif ($header) {  ## Match search string
            $data .= $_;
            $match = 1;
        }
    }
}

sub case2 {
    open(my $fh, '<', "$fileName") or die $!;
    my $match; my $data; my $header;

    while (<$fh>) {
        if ($_ =~ $headerRegex) {
            if ($match) {
                $data =~ s/$greenRegex/\033[1;32m$1\033[m/o;
                $data =~ s/$redRegex/\033[1;31m$1\033[m/go;
                $data =~ s/\n\n/\n/go;
                print $data, "\n";
                $match = 0;
            }
            $data = ""; $header = 1;
        }
        if (($header) && (0 > index ($_, $string))) {
            $data .= $_;
        } elsif ($header) {
            $data .= $_;
            $match = 1;
        }
    }
}

case1 4.79 s/iter
case2 46.6 s/iter
[download]

Yes I apologize it is 15 seconds for my grep method vs 50 seconds for doing it all in Perl as you've shown. I must have done something weird before to increase the time by so much.