Hello msh210,

Ah, so it's the loading of the entire file that's slowing me down. And you say that your and kcott's solutions don't avoid that.

No, Anonymous Monk did not say that! Quite the contrary: he and kcott are recommending that you read the file line-by-line, stopping when the first match is found. (kcott’s quotation from the Tie::File documentation specifically rules out any need to read in the whole file.)

Of course, this assumes that if a match occurs at all, it will occur within a single line. If your matches may overlap two or more lines, you will need to adapt the approach by employing a sliding window technique to examine n lines at a time, moving the window forward each time by discarding the first line and adding the next (n + 1th) line to the window. Then the key task is to determine the optimum size of n — which must be large enough to ensure that all possible matches are accommodated. To find the most efficient size for n, a certain amount of trial-and-error is usually required.

Hope that helps,

Ah, I hadn't understood. Thank you.

$_="msh210";$"=$\;@_=@{[split//,uc]}[2,0];$_="@_$\1";$\=$/;++$_[0]for$...1;print lc substr crypt($_,"@_"),1,6

Ah, so it's the loading of the entire file that's slowing me down.

And you say that your and kcott's solutions don't avoid that

use strict;
use warnings;

my @strings = qw(
  foobarbaz111222333444
  barbazfoo111555666999
  bazfoobar999888777666
);

my $text = <<'END';
bdfjadslfjhasdjklfhjklashdflkjadshfjkladhfjkldhfljkafjadfdji
adlfjhdlsjkfuiowerfhwehrbeblbflasdfbhjkaldhqpqheuihfbdfkhyyy
qjdpdbnfdbjdfklbasjfbajksdbfjaksdbfjaksdfbjkasdfhydsfadfjyyy
END

$text x= 1_000_000;
$text .= "bazfoobar999888777666\n";

printf "text size: %d bytes (%.2f mb)\n",
  length($text), length($text) / ( 1024 * 1024 );

my $regex = join '|', map quotemeta, @strings;
$regex = qr/^($regex)$/m;

printf "found %s at %d\n", $1, $-[0] if $text =~ $regex;
[download]

[I haven't logged in for a tenday or so. Apolgies for the late response.]

In an earlier post, Anonymous Monk wrote:

"The approach suggested by kcott (compiling the combined regex; not the tie stuff) should be fairly efficient."

I'd just like to say that I concur with the "not the tie stuff" part.

"... to address some of the questions in kcott's reply ..."

Thanks for that. I think you shouldn't use any modules at all (except the pragmata). Consider code like this:

#!/usr/bin/env perl
#
#   pm_1155868_first_grep_large_file_2.pl [no modules]
#

use strict;
use warnings;
use autodie;

open my $lines_fh, '<', input_filename();
while (<$lines_fh>) {
    next unless is_a_match($_);
    process_match($_) and last;
}

sub input_filename { 'pm_1155868_input.txt' }
sub get_strings { [qw{rst uvw xyz}] }
BEGIN {
    my $re = '(?:' . join('|', @{get_strings()}) . ')';
    sub is_a_match { $_[0] =~ /$re/ }
}
sub process_match { print "Match: $_[0]" }
[download]

It produces the same output:

$ pm_1155868_first_grep_large_file_2.pl
Match: dddddddxyzdddd
[download]

• You'll need to modify &input_filename, &get_strings and &process_match to suit.
• $re and &is_a_match should work fine as is; modify if necessary.
• Note how the while loop ignores all records that don't match (next); when the first match is found, it uses it and then exits the loop (last). This seems to match your requirements.
• Use Benchmark, as previously discussed, to compare potential solutions.