Dear Monks

The problem I need to solve is as follows:

I have an unknown number of optimized (assembled) regular expressions stored in separate files. Now I want to parse one big file and compare each line to all of those regular expressions. If a match is found, the line should be written to a file that corresponds to the regex. E.g. if I have 20 regex, I should end up with 20 Files. The reason I want this to happen simultaneously is, that the big file is really big (~10GB), so I don't want to go through it 20 times (however this might even be faster, as I fear, the filehandling withing the while(<>) loop eats up all the performance).

My approach was to readdir and push all regex to an array, as well as all the filehandles.

foreach(@inputs) {
    local *FILE;
    $file = "$FindBin::Bin/../rxo/$_";
    $outfile = "$FindBin::Bin/../blocks/$_";
    open(FILE, "$file") || die; 
    open(OUTFILE, ">> $outfile") || die; 
    $fh = \*FILE;
    $ofh = \*OUTFILE;
    $regex = <$fh>;
    push(@regex,$regex);
    push(@filehandles,$ofh);
    }
[download]

That looks ok, now the tricky part is to parse the big file, I tried the following:

my $i = 0;
while(<>) {
  foreach(@inputs) {
  print @filehandles[$i] if (/@regex[$i]/../^END_OF_BLOCK/);
  $i++;
  }
  $i=0;
}
[download]

This runs for 12 hours now but should take between 2 and 6 hours according to my calculations. Furthermore it didnt write anything to a file yet (maybe this happens not before the script is done and filehandles are closed). I think one problem is, that the filehandling stuff is inside while(<>) or the foreach() slows down the whole thing. Also I am not even sure, my print-line is correct. Notice, that I not only want to save the matching line but the complete block of lines starting with the regex match and ending with a "end of block" line. Furthermore I believe there are more elegant ways to solve the problem(s). Thanks in advance for any thoughts!

Update/Solution

There were several problems with my code and I finally was able to figure out all of them with your help

• Withing while I had to read the input into $line and not use $_
• I had to wrap curly braces around my $filehandle$i, however I changed this code to the suggestion of BillKSmith later
• I formerly tried to read blocks between two regex in the style /start/../end/ which seems not to work with the construct of $line instead of $_ so I switched to the suggestion of changing the input record separator $/ to the end of block statement
• By using a new $/ my start-regex (starting with a "^" beginning of line statement) were not correct any longer because between end-regex and start-regex there may occur other lines.
• Also my output filehandler was wrong, always pointing to the last filehandler

my final code looks like this

#!perl

use strict;
use warnings;
use FindBin;

my $dir ="$FindBin::Bin/../rxo";
opendir(my $dh, $dir) || die "can't opendir $dir: $!"; 
my @inputs = readdir($dh);
closedir $dh;
splice @inputs, 0, 2;

my @dispatch;
foreach(@inputs) {
    my $outfile = "$FindBin::Bin/../blocks/$_";
    #open(FILE, "$file") || die; 
    open my $ofh, '>', $outfile || die; 
    my $file = "$FindBin::Bin/../rxo/$_";
    open my $fh, '<', $file || die;;
    my $regex = <$fh>;
    close $fh;
    push @dispatch, { file => $ofh, regex => $regex };
    }

while(my $line = do { local $/ = 'THE_END'; <> }) {
    foreach (@dispatch) {
        print { $_->{file} } $line if $line =~ $_->{regex};
        print $line if $line =~ $_->{regex};
        }
}
[download]

this can be mark as solved. thanks a lot to everyone!

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.