Are you saying writing a mod_perl handler will be a better performance boost than anything that can be done to the mapper script?

Yes, I believe that to be the case, simply by looking at all the work mod_rewrite has to go through to talk to your script, compared to just executing a simple bit of code already compiled into the on-board Perl interpreter.