Indeed, you will only see the speed-up the second time the script runs on that interpreter.

As a rule your scripts should run under standard CGI before you try mod-perl as there are already enough pitfalls when using mod-perl, that you do not need the errors in your script on top of that.

As a matter of fact, when you run your script from the console it does get compiled before it is run: but when the script ends Perl wipes it memory so next time you start afresh. Only mod-perl is able to "remember" the compiled script.