$& slows down every regular expression in your program where $1 and friends just slow down those regular expressions that use them. Same goes for $` and $'. This is documented (albeit sparsely) in perlvar