Hmm, no mention of Perl in your question, marking a post (OT) would be nice :) I'd suggest reading http://www.robotstxt.org/, and How do I prevent robots scanning my site? which details your current solution with the caveat:

"but this only helps with well-behaved robots."

and a reference to Can I block just bad robots?

Hope this helps,

Martin