• (...) should be ( (?:...)* ).
• [^()]+ should be [^()"]+.
• $1 will contain the outer match, not the inner one.
• The s modifier is useless since you don't have any . in that regexp.
• The s modifier would also be useless on the qr//, which is not affected by the existing s.
• It wouldn't hurt to anchor the regexp to the start of the string.

use strict;
use warnings;

my $textInner =
  '(outer(inner(most "this (shouldn\'t match)" inner)))';

my $innerRe;
$innerRe = qr/
                \(
                (
                   (?:
                      [^()"]+
                   |
                      "[^"]*"
                   |
                      (??{ $innerRe })
                   )*
                )
                \)
             /sx;

$textInner =~ /^$innerRe/g;
print "outer: $1\n";
[download]

(Sorry, I took out the comments to debug. Feel free to re-add them.)