There is a penalty for $&, but it's not as bad as it used to be in olden times. Running some benchmarks (the $& one in a separate perl to avoid tainting the non-$& results):

$ perl -MBenchmark=cmpthese -le 'my $a; $_ = "abcdefghi"; cmpthese( 2_
+000_000, { "01 \$&" => sub { /def/; $a = $& } } )'
           Rate 01 $&
01 $& 1025641/s    --
$ perl -MBenchmark=cmpthese -le 'my $a; $_ = "abcdefghi"; cmpthese( 2_
+000_000, { "02 substr" => sub { /def/; $a = substr( $_, $-[0], $+[0] 
+- $-[0] ) }, "03 paren" => sub { /(def)/; $a = $1 } } )'
               Rate 02 substr  03 paren
02 substr  116822/s        --      -89%
03 paren  1098901/s      841%        --
[download]

So using $& is only about 4-7% slower than capturing parens (I did several runs of these and this is one of the slow results sets), and both of them are about ten times as fast as the substr() incantation (presumably since $&/$1 are happening all in C under the hood rather than going back and forth a bit as it does).

So yeah be aware there's a hit, but don't let it scare you away from $& completely.