Testing regex equivalence

loris has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: Testing regex equivalence by Corion (Patriarch) on Feb 21, 2006 at 12:57 UTC
I'm not sure what exactly your goal is. Every regular expression of a certain size either has no loops or it has loops, and if it has loops, there is a certain string length after which parts of the regular expression repeat (I believe this is more or less the pumping lemma). So to find out whether two regular expressions are equal, you need to check all strings below the maximum of the two respectice string lengths. But this idea only works for "real" regular expressions (in the computer science meaning), not for Perl Regular Expressions in general. So it will help to know if your regular expressions are regular or not. If they are regular in the CS sense, you can parse the regular expression and then convert it into a finite state machine. Then you "just" need to create all strings of length smaller or equal to twice the number of states, and check for those strings if they either match both or fail both regular expressions.	[reply]
Re^2: Testing regex equivalence by loris (Hermit) on Feb 21, 2006 at 13:13 UTC
What I want to do is really to parse the regex in some way and simplifiy it, if possible. So say I have a regex generator and for a certain input it produces `.(.)$` I would like to be able to recognise that this is the same as `.*` Thanks, loris "It took Loris ten minutes to eat a satsuma . . . twenty minutes to get from one end of his branch to the other . . . and an hour to scratch his bottom. But Slow Loris didn't care. He had a secret . . ."	[reply] [d/l] [select]
Re^3: Testing regex equivalence by Corion (Patriarch) on Feb 21, 2006 at 13:19 UTC
They aren't strictly equivalent, because the first regex has capturing parentheses even if they only will ever match the empty string at the end of the string. But assuming you have `/(?:.).$/ # and /.$/` [download] you could convert them into canonical form, that is, normalize all `.` to become `(?:.)`, and then simplify, mapping `(?:.)(?:.)` to `(?:.)`.	[reply] [d/l] [select]
Re: Testing regex equivalence by blokhead (Monsignor) on Feb 21, 2006 at 17:25 UTC
See also this thread. The short answer is that you can do this by converting regular expressions to NFAs/DFAs (for more details, see my reply in the thread I referenced). However, this cannot be done in a super-efficient way. In fact, the problem of deciding equivalence of two regular expressions is PSPACE-complete (which is bad ;)). Even in your special case of deciding whether a regular expression accepts all strings, the problem remains PSPACE-complete (google for regular expression universality). Being PSPACE-complete does not mean it's impossible. It's quite possible, but will quickly become very impractical for large regular expressions.. As for how to do the conversion to NFA/DFA, I can tell you that another monk and I are working on a formal language & automata toolkit that will make this easy, but it won't be ready for a while... Update: as for your idea above to "simplify" a regex, that is essentially the same thing and will not gain you anything. Testing regex equivalency reduces to the problem of finding the "simplest" equivalent regex (no matter how you define "simplest"), so regex simplification is still a PSPACE-hard problem (-hard instead of -complete because we're now talking about a search procedure, not a decision procedure). blokhead	[reply]
Re: Testing regex equivalence by aquarium (Curate) on Feb 21, 2006 at 13:07 UTC
you could use "b" or whatever it is to compile a perl regex into token form. equivalent regular expressions (e.g. /.../ and /.{3}/ should end up the same. as for more complicated regular expressions, they may end up being equal depending on the dataset being fed in. the only way to test this kind of equivalence is to run both on the dataset in question. the hardest line to type correctly is: stty erase ^H	[reply]
Re^2: Testing regex equivalence by grinder (Bishop) on Feb 21, 2006 at 15:25 UTC
equivalent regular expressions (e.g. /.../ and /.{3}/ should end up the same Evidently, you did not try this out yourself before posting. It took me 10 seconds to see that % perl -Mre=debug -e '/.../' Freeing REx: `","' Compiling REx `...' size 4 Got 36 bytes for offset annotations. first at 1 1: REG_ANY(2) 2: REG_ANY(3) 3: REG_ANY(4) 4: END(0) minlen 3 Offsets: [4] 1[1] 2[1] 3[1] 4[0] Freeing REx: `"..."' [download] and % perl -Mre=debug -e '/.{3}/' Freeing REx: `","' Compiling REx `.{3}' size 4 Got 36 bytes for offset annotations. first at 3 1: CURLY {3,3}(4) 3: REG_ANY(0) 4: END(0) minlen 3 Offsets: [4] 2[3] 0[0] 1[1] 5[0] Freeing REx: `".{3}"' [download] are quite different beasts. So that's not going to fly. Unfortunately, I don't have any better ideas myself. The only thing I can think of would be to take a look at japhy's Regexp::Parser, which will ease the pain of picking a pattern apart, but to put them back together again with an eye to checking for equivalence will be non-trivial. • another intruder with the mooring in the heart of the Perl	[reply] [d/l] [select]
Re: Testing regex equivalence by mickeyn (Priest) on Feb 21, 2006 at 12:45 UTC
can you post the regex you wish to check ? Mickey	[reply]
Re^2: Testing regex equivalence by loris (Hermit) on Feb 21, 2006 at 12:50 UTC
I don't want to check a specific regex. I want to check a large number of them. Thanks, loris "It took Loris ten minutes to eat a satsuma . . . twenty minutes to get from one end of his branch to the other . . . and an hour to scratch his bottom. But Slow Loris didn't care. He had a secret . . ."	[reply]