I wouldn't worry at all about performance between the two. Just use what makes sense in your code. Don't do stuff like this:

$some_condition ? &some_sub : &other_sub;

if ($some_condition) {
   $some_var = &something;
} else {
   $some_var = $something_else;
}
[download]

Use the operators for what they were designed, and your code will be infinitely more readable and usable.