You are making the regular expression engine do a lot of extra work by:

1) Using capturing parenthesis instead of noncapturing ones .
2) Even a non-greedy dot-star is bad (see Death to Dot Star!). Use a negated character class like [^\]]* instead.
3) Excessive going in and out from parenthesis levels, as in the [^\[] expression following the "|", which ought to have a "+" after it.

These inefficiencies are probably causing the regular expression engine to go wild and run out of memory. By the way, why not use module Text:Balanced instead?

Update: Here is a brushed-up version with my suggestions added: 

if ($text =~  /^
                (?:                  # Any amount of:
                  (?:
                    \[              # An open brace
                     [^\]]*         # Any amount of non-brace stuff
                    \]              # A close brace
                  )
                  |                 # Or
                  [^\[]+            # Anything that's not an open brac
+e.
                 )*
                 \]                 # Followed by a close brace
              /xs

[download]

Update2: After recommending Text::Balanced for this problem, I decided to try it for myself. I would have expected the following code to work, but it doesn't (I get the error: and it does work as long as you test for the following message as an acceptable case: "Did not find opening bracket after prefix: "[^\[]*", detected at offset 1216". The message just means you have passed all the brackets, which is fine. 

my $next;
while ( $next = (extract_bracketed($text,'[]','[^\[]*'))[0] )
{
   print "found matching brackets: *$next*\n";
}
print "found bracket error: $@\n" if $@;

[download]

In reply to Re: OS X vs. My Regex by tall_man
in thread OS X vs. My Regex by Pedro Picasso

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.