Paired delimiters can nest. Hence: my $exp = qr(x(capture)y); is valid. Perl recognizes the parens inside the expression as being inside the expression. In the example you gave, there is a ( inside the expression before the ) inside the expression, so Perl understood it as a nested pair.

You can see how perl is parsing the regex by printing it:

>perl -e "print qr(a(b)c)"
(?-xism:a(b)c)
>perl -e "print qr(a\(b\)c)"
(?-xism:a(b)c)
>perl -e "print qr(a\\(b\\)c)"
(?-xism:a\\(b\\)c)
[download]

As you see, a single backslash is interpreted as escaping the parens from the qr(), rather than as part of the regex. A double-backslash escapes the backslash in the regex, so there's no backslashy way to match a literal paren. You have to use a character class, plus a backslash, unless you're going to have a balancing paren in the expression, in which case, perl will notice the balanced parens and Do The Right Thing.

>perl -e "print qr(a\(bc)"
Unmatched ( in regex; marked by <-- HERE in m/a( <-- HERE bc/ at -e li
+ne 1.
>perl -e "print qr(a[(]bc)"
Search pattern not terminated at -e line 1.
>perl -e "print qr(a[\(]bc)"
(?-xism:a[(]bc)
[download]