...using variables like $1, $2 will not really improve performance...

No the capture vars ($1, $2 etc.) aren't the culprits.

From perlretut:

If you use them, perl will set $` to the part of the string before the match, will set $& to the part of the string that matched, and will set $' to the part of the string after the match.
...
It is important to note that using $` and $' slows down regexp matching quite a bit, and $& slows it down to a lesser extent...

So $1 and friends won't effect performance.