As Fletch mentioned, the trick is to use the (?{code}) assertion. Increment the current depth when entering the recursion (left parenthesis in the example), decrement it when leaving (right parenthesis). When incrementing, store the maximum.

#!/usr/bin/perl
use warnings;
use strict;
use feature qw{ say };

my $string = '((()))(((())))(()(()))';

my $max = 0;
my $depth = 0;

if ($string =~ /^(
                    \(
                    (?{$max = $depth if ++$depth > $max})
                    (?1)*
                    \)
                    (?{--$depth})
                )
                (?1)*
                $/gx
) {
    say $max;
}
[download]