Dear Monks,

today I re-stumbled upon an issue I quick-and-dirty solved a while ago but I want to solve it more elegantly while I am doing code polishing these days.

I use Regexp::Assemble to assemble regex that are about 15kb to 87kb large. Now I very simply run through a large (~10GB) file and match the regex. I used to do this on the command line in the style

perl -ne 'print if (/MYLARGEREGEX_HERE/../END_OF_BLOCK/)' inputfile > 
+outputfile

[download]

this was fast as hell. However when my regex grew in size, I was not able to copy paste them into the bash so I started to read the regex from a file and did something like this

#!perl

use strict;
use warnings;

open my $fh_big_file, '<', $ARGV[0] || die; #first argument must be th
+e input file
open my $fh_regex, '<', $ARGV[1] || die; # second argument points to t
+he file containing the regex
my $regex = <$fh_regex>;


while(<$fh_big_file>) {
  print if (/$regex/../^END_OF_BLOCK/);
}

[download]

The funny thing is, that this flavour of code costs me factor 20 in speed or even more. I can reclaim the speed by avoiding to store the regex in a variable, e.g.

while(<$fh_big_file>) {
  print if (/MY_HUGE_REGEX_JUST_PLAIN/../^END_OF_BLOCK/);
}

[download]

so I assume this has something to do with fetching the content of the variable (from RAM to CPU?) over and over for each loop of while(<>), whereas inputing the regex directly doesnt need to re-read it every time.

This approach however requires me to manually copy the regex to its place each time I run the whole procedure of "assembling, searching, processing, assembling, seachring, processing" and I would like to automize it without loss of performance. Any ideas?

thanks and cheers!

Update/Solution/Close

The suggestion to use the o operator works. However it needs to be behind /$regex/ not behind /END_OF_BLOCK/. i.e. like shmem suggested:

while(<$fh_big_file>) {
  print if (/$regex/o .. /^END_OF_BLOCK/);
}

[download]

thanks!

In reply to Performance of assambled regex by Foodeywo

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.