You can use substr to constrain the part of the string searched (without replication). And by using the offset of the end of the last capture group you can move the window along the string efficiently:

$s = 'abc' x 10; ## repetative test data
$p = 0;          ## start at offset 0

## match against the substring starting at the offset
## and capture (3) items
while( my @x = substr( $s, $p ) =~ m[(.)(.)(.)] ) { 
    print "@x"; ## do something with the captures

    ## and advance the offset to the end of what was matched
    $p += $+[3]; 
}

a b c
a b c
a b c
a b c
a b c
a b c
a b c
a b c
a b c
a b c
[download]

When the while loop exits, the remainder beyond the offset could contain a partial match, so delete the front of the string and append the next read to the end.

Yes!

It was something like this I was looking for. Only, since my regex get passed in from the user (and I therefore I cannot know the number of parethesized subexpressions used), I need to advance my offset with $+[0];.

Could someone possibly refer me to some point in the docs that states that this kind of use of substr really is efficient? (I know I too have read that somewhere, sometime long ago -- but where?)

Surprised again by what is efficient/inefficient in perl.

Turns out that trying to keep track of submatches simply isn't worth it. If I split my log using a regex (with the above mentioned parethesized subexpressions whose content I wish to retain and use too) to match the head of each log entry, it is simply many times faster to apply the regex again to each individual log entry to get the list of matching subexpressions, than it is to store them in an array-of-arrays and pass them around for later reference.

My guess is that it is rather expensive to toss references to lists around, while matching a regex against a small text (a single log entry) where you know it is going to match at the first character position, is cheap.

While this double-regex matching seem redundant, it has the benefit of making the program both fast, and easy to read. I'll get back to ya'all with the code soon enough. :)

Ahh, but I'm also looking to obtain any subpatterns (paranthethized expressions) inside my regex. Putting @x there was ment to capture them.

I naïvely thought m//g would return the same kind of stuff as m// without the /g flag, so that I could iterate over the string, obtaining one array of subpatterns at a time.

But m// resets pos() and always begin at the start of the string, so that won't do either.

Is there some way I could get m// to start searching where the last match left off? Using substr()? How efficient is that? Does it make a new copy of the string, or operate directly on the original?

"If you don't want the position reset after failure to match, add the //c, as in /regexp/gc."

while (@x = /($regex)/gc) {
   push @buffer, [ @x ]; # Or, if you localize (my) @x, push @buffer, 
+\@x ;
}
[download]

No. That won't work. The loop will only be executed once because of the /g flag, which causes all matches to be returned at once.

Sorry. :(

Good catch.

OK - here is a somewhat contrived variation that seems to work for me:

use strict;

$_="this  that other thing that thw tests sometimes";
my $regex = qr/(t..)\w+\s+?(\w\w)(.+)/;
my  @buffer;
my @x;
while ( (@x[0..1],$_) = /$regex/) {
   push @buffer, [ @x ]; 
   print join(",",@x)  . ";\n"; 
}
[download]

Changes:
• You need to know how many elements to expect
• The regex has been modified to return all remaining information as the final element
• The final element gets re-stored into $_
• No more /g modifier

     "As you get older three things happen. The first is your memory goes, and I can't remember the other two... " - Sir Norman Wisdom

Can you reset the IRS $/, that might be much easier than using sysread which might break down your records. If you do need that AOA, you can maintain a queue in fixed-size or limited-size or whatever.

BTW. be very caution when using regexes with large chucks of arbitrary texts.

Regards

Lihao(XC)