Nothing in your code jumps out but whenever you get weird, inexplicable delays check that there's not something reliant on a network connection (locally or to the outside world). A not completely synthetic example; I've had something similar happen: if you have something trying to resolve a hostname (for whatever reason) and your DNS is intermittently unavailable or timing out you can get inconsistent runtimes from that. Also you want to make sure you're using your shell's builtin time command to get execution time of just your script. If you're doing something like date ; perl mytest.plx ; date and then manually computing the time there could be something (maybe your shell has a prompt command that does something network-y) that's making you see a delay but it's not in your code.
The cake is a lie.
The cake is a lie.
The cake is a lie.