Assuming that what is inside &my_sub is optimized, I can't think of much you can do. According to my (limited) research, passing a scalar by reference doesn't typically buy you any speed. Unless, I suppose, the scalars are huge.

You may consider something like the following, assuming that &my_sub returns a value:

sub my_sub($) {
    ... do something EASY and fast ...
    return \$foo
}
foreach my $i (0 .. $a_bazillion) {
    $return_value = ${my_sub($some_value);
}
[download]

While I've never tried this personally, it may make a difference regarding performance. I suggest liberal use of use Benchmark; and see what helps.

HTH!