So I was trying to debug a problem I was having with a regular expression in that other language I use professionally and thought "well, let me see if perl's regular expression engine behaves the way I think it should on this". So then I copy it over and, while perl doesn't have the majorly unpleasant O(2**n) time behavior I was seeing with java.util.regex, it does misbehave in a bizarre fashion. Or at least, I think it does. I've narrowed down the test case as much as I can to still see the strange behavior.

The paired-down code attempts to split incoming lines on XX, but only when XX isn't inside a single-quoted string: 

#!perl

while (<DATA>)
{
  chomp;
  print "for [$_]:\n";
  # split on XX, but only when it's not in
  # single quotes.
  while (m% ((?:[^']*?         # Some unquoted stuff
                 (?:'[^']*')?  # optional quoted stuff
              )*?)             # many times, but non-greedy
           XX |                # field separator, or
            (.+) %xg)          #   a non-blank final field
  {
    # field value should now be in defined($2)?$2:$1
    my ($t1, $t2) = qw(undef undef);
    $t1 = "[$1]" if defined($1);
    $t2 = "[$2]" if defined($2);
    print "\t$t1\t$t2\n";
  }
}
__END__
aXXbXXc
abcd
a little 'quote XX' quote stuff XX other

[download]
Now here's where it gets weird. The trailing fields are coming out one character at a time. Here's the output: 
for [aXXbXXc]:
        [a]     undef
        [b]     undef
        undef   [c]
for [abcd]:
        undef   [a]
        undef   [b]
        undef   [c]
        undef   [d]
for [a little 'quote XX' quote stuff XX other]:
        [a little 'quote XX' quote stuff ]      undef
        undef   [ ]
        undef   [o]
        undef   [t]
        undef   [h]
        undef   [e]
        undef   [r]

[download]

It's as though perl had decided that my last + sign in the regular expression should be non-greedy despite the fact that it's not followed by a ?.

What's going on here? It gets even more bizarre:

If I replace the first half with a pattern that really should be equivalent, this behavior goes away: 

#!perl

while (<DATA>)
{
  chomp;
  print "for [$_]:\n";
  # split on XX, but only when it's not in
  # single quotes.
  while (m% ([^']*?         # Some unquoted stuff
             (?:'[^']*'     # optionally, quoted followed by 
                 [^']*?)*? )# unquoted stuff.  non-greedy
           XX |             # field separator, or
            (.+) %xg)       #   a non-blank final field
  {
    # field value should now be in defined($2)?$2:$1
    my ($t1, $t2) = qw(undef undef);
    $t1 = "[$1]" if defined($1);
    $t2 = "[$2]" if defined($2);
    print "\t$t1\t$t2\n";
  }
}
__END__
aXXbXXc
abcd
a little 'quote XX' quote stuff XX other

[download]

To summarize: I have a regular expression match that is of the form m%(foo)XX|(.+)%g, where foo is a slightly complicated expression with no captures. When I run it, I get single character results repeatedly in $2. When I replace foo with a different complicated expression that should be equivalent, I suddenly get multiple characters in $2.

I've verified this behavior with cygwin's 5.8.6 perl and ActiveState's 5.6.1. (build 635) 

--
@/=map{[/./g]}qw/.h_nJ Xapou cets krht ele_ r_ra/;
map{y/X_/\n /;print}map{pop@$_}@/for@/

[download]

In reply to This regex seems to have splattered non-greedy everywhere by fizbin

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.