And this is why you need production and dev boxes. In the production box, you only need enough logging so as to know when something breaks. In your dev box, you can re-harvest the offending pages and see the errors in all its glory, including the ability to add instrumentation to the code on the fly.

If you cannot have the two boxes, perhaps you can run a second instance of your spider manually, when needed. I bet this is less resource-intensive that managing such huge logs.

That being said, take a look at Logfile::Rotate.