RFC: Regexp::AllMatches

The problem

Some years ago I wrote a class to find all possible matches of a pattern against a string, including overlapping matches. If there was an /a switch that accomplished this, it would look like this:

    foreach ('abc' =~ /.+?/a) {
        print "$_\n";        
    }

    __END__
    a
    ab
    abc
    b
    bc
    c
[download]

I figure I'm going to release it to CPAN. Before I do that I'd appreciate some feedback.

Description of the classes

I currently call it Regexp::AllMatches and use an OO interface. Here's how it's used:

    use Regexp::AllMatches;

    my $matcher = Regexp::AllMatches->new(STRING => qr/PATTERN/);

    while (my ($match) = $matcher->next) {
        print "$match\n";
    }
[download]

$matcher is a simple iterator, and the only methods are

    * new
    * clone
    * next
[download]

$match is a match object that stringifies to the matched string ($&) and implements the following methods:

    * prematch ($`)
    * match ($&)
    * postmatch ($`)
    * group ($<*digits*>)
    * groups
[download]

I also wrote Regexp::AllMatches::Extended that implements some extra convenience methods at the cost of memory and speed.

    * curr
    * prev
    * reset
    * all
[download]

Regexp::AllMatches and Regexp::AllMatches::Extended will be two different modules. The match object is currently defined in Regexp::AllMatches, and is at the moment not for public instantiation.

So, what do you think of

the class names?
the overall design?

Any feedback and thoughts will be appreciated.

lodin

Update: Regexp::Exhaustive is the new name. I'm not too happy about Regexp::Exhaustive::Extended though. Any ideas? How about Regexp::Exhaustive::Extra(s) or Regexp::Exhaustive::Convenient?

Update: After a bit of cleaning Regexp::Exhaustive::Extended became nothing but a generic iterator decorator, so it's gone. The all method is now put directly in Regexp::Exhaustive instead.

Update: Uploaded to CPAN as Regexp::Exhaustive.

Comment on RFC: Regexp::AllMatches Select or Download Code

Replies are listed 'Best First'.
Re: RFC: Regexp::AllMatches by moritz (Cardinal) on Aug 06, 2007 at 17:06 UTC
In Perl 6 that's the `:ex` or `:exhaustive` modifier, maybe you could consider calling your module Regexp::Exhaustive or Regexp::Match::Exhaustive or something. The API looks nice, I just wonder why you use the parens in `while (my ($match) = $matcher->next)` - does `->next()` return a list? if yes, why? Update: while rereading S05 I also found the `:overlap` modifier - please check if you are implementing :exhaustive or :overlap - I think it's :exhaustive, but I'm not sure. Perl 6 in German	[reply] [d/l] [select]
Re^2: RFC: Regexp::AllMatches by lodin (Hermit) on Aug 06, 2007 at 17:30 UTC
I'll definately consider ::Exhaustive. Thanks! does `->next()` return a list? if yes, why? There are different opinions and tastes regarding this. I prefer to design iterators so that the next method returns `undef` in scalar context and the empty list in list context when it's exhausted. A successful match may be "" or "0", and that would force me to write `while (defined(my $match = $matcher->next)) { ... }` [download] to not leave the loop prematurely. lodin	[reply] [d/l] [select]
Re^3: RFC: Regexp::AllMatches by wind (Priest) on Aug 07, 2007 at 00:25 UTC
There is another way to handle the return value in a loop. Instead of returning a string, return an object that overloads the boolean, numerical, and string operators. An example of this in practice is the IO::Prompt module on cpan. Here is an excerpt of the source: `package IO::Prompt::ReturnVal; use overload q{bool} => sub { $_ = $_[0]{value} if $_[0]{set_val}; $_[0]{handled} = 1; $_[0]{success}; }, q{""} => sub { $_[0]{handled} = 1; "$_[0]{value}"; }, q{0+} => sub { $_[0]{handled} = 1; 0 + $_[0]{value}; }, fallback => 1, ; sub DESTROY { $_ = $_[0]{value} unless $_[0]{handled}; }` [download] As you can see, it also provides mechanisms for setting $_. Anyway, just wanted to throw out another option. Please note that the IO::Prompt module only runs on unix based systems currently AFAIK. - Miller	[reply] [d/l]
Re: RFC: Regexp::AllMatches by blokhead (Monsignor) on Aug 07, 2007 at 02:07 UTC
I'm interested about this being implemented as an iterator. I can imagine two ways this might be done: You exhaustively generate all matches, and the iterator is just added on to have a nice interface. Often if you see an iterator interface you assume that it is only generating the return values "on demand" and not pre-computing them all. The iterator is implemented by somehow jumping out of a regex during a match with lots of back-tracking. If this is the case, I wonder if re-entrancy would be a problem. Could the regex engine be used while this iterator object is "active?" Actually, implementing an iterator instead of a callback in this way seems highly non-trivial to me, so if this is the case, I'd be interested to see the implementation details. Either way, I think it would be nice if the documentation made clear what was going on with respect to iterators. I like how the interface provides a way to get the $1, $2, etc match variables. blokhead	[reply]
Re^2: RFC: Regexp::AllMatches by lodin (Hermit) on Aug 07, 2007 at 02:37 UTC
Often if you see an iterator interface you assume that it is only generating the return values "on demand" and not pre-computing them all. ... and that is also true here. Since backtracking patterns quickly generate a very large number of matches I don't dare to precompute them. Could the regex engine be used while this iterator object is "active?" The following code works, and that makes me believe there won't be any other re-entrancy issues. But I know very little about the internals of the engine, and what may blow under certain circumstances. `use Test::More 'no_plan'; use Regexp::AllMatches; my $str = 'abc'; my $m1 = Regexp::AllMatches::->new($str => qr/.+/); is($m1->next, 'abc'); my $m2 = $m1->clone; is($m1->next, 'ab'); is_deeply([ $str =~ /./g ], [qw/ a b c /]); is($m1->next, 'a'); is($m2->next, 'ab'); __END__ ok 1 ok 2 ok 3 ok 4 ok 5 1..5` [download] lodin	[reply] [d/l]