A single character can make a huge difference in how large of a match your regex is able to make (it likely also makes your regex more efficient):

use re 'debug';
my $braces = qr/(?<braces>\{ ([^\{\}]* | (?&braces))*? \} )/x;
#                                    ^

Compiling REx "(?<braces>\{ ([^\{\}]* | (?&braces))*? \} )"
Final program:
   1: OPEN1 'braces' (3)
   3:   EXACT <{> (5)
   5:   MINMOD (6)
   6:   CURLYX[0] {0,32767} (30)
   8:     OPEN2 (10)
  10:       BRANCH (23)
  11:         STAR (27)
  12:           ANYOF[\0-z|~-\377][{unicode_all}] (0)
  23:       BRANCH (FAIL)
  24:         GOSUB1[-23] (27)
  27:     CLOSE2 (29)
  29:   WHILEM[2/1] (0)
  30:   NOTHING (31)
  31:   EXACT <}> (33)
  33: CLOSE1 'braces' (35)
  35: END (0)
anchored "{" at 0 floating "}" at 1..2147483647 (checking floating) mi
+nlen 2 
Freeing REx: "(?<braces>\{ ([^\{\}]* | (?&braces))*? \} )"
[download]

You still have an instance of {0,32767}, but it means you can match 32K instances of braces that are separated by non-brace runs where each run can be huge (note how the new * was translated to STAR w/o 32K mentioned and not to CURLYX).

Your prior regex would have to recurse into the 'braces' construct for every single new character matched. Since it would only do that upto 32K times, you could only match about 32KB of text.

Update: I should have added a + not a *.

Warnings enabled, the 32k problem should have manifested itself:

Complex regular subexpression recursion limit (32766) exceeded

my $braces = qr/(?<braces> { (?: (?: [^{}] | (?&braces) )*+ )* } )/x;
[download]

perlre has the following example on handling nested parens:

my $parens = qr/(\((?:[^()]++|(?-1))*+\))/;
if (/foo $parens \s+ \+ \s+ bar $parens/x) {
   # do something here...
}
[download]

Thanks for all the replies! I tried adding the following test to see if I can get it working (as I said, I have now rewritten it to do the parsing by hand, but I would like to figure it out for next time):

 
$braces = qr/(?<braces>\{ ([^\{\}]++ | (?&braces))*+ \} )/x;
if ($h =~ s/class \s+ (?<class_name> \w+) \s* (\: (\s* \w*)? \s* (?<an
+cestor> \w+))? \s* $braces//x) {
   print "hi\n";
}
[download]

The match is never made however... The code I'm matching against is a C++ header file, and I'm looking for class definitions, formatted like this :

class MyClassName : public MyAncestorClass {
...
...
};
[download]

This expression works fine for smaller classes or for this same problem class if there are no braces inside the class definition. In this class however, I have the following declarations (C++ Builder code...) :

  __property TNotifyEvent OnMonaDocSaveToDB = {read=FOnMonaDocSaveToDB
+, write=FOnMonaDocSaveToDB};
[download]

If I remove that property declaration, it matches, if I leave it, it fails... Jonathan

I just checked, there is no warning when I execute my script using "perl -w"...

On the off chance that you've stumbled upon an undiscovered bug, the considerate route to take is to provide a testcase. This may involve some tedious work reducing a self-contained test unit to a minimum. That, of course, after you've verified the behavior with the latest perl.