I'd upload a new version, with an email address you don't use for anything else. Make it clear in the documentation you're no longer supporting the module, and that all bug reports will be ignored. Redirect any incoming mail on the email address to /dev/null.

Unfortunally, that will not be the end of it. CPAN has a no-opt-out RT system for all modules. You will be getting RT tickets for your module (as they will be send to the address you have registered for your CPAN account). Mark Lehmann can tell you all about it -- his rants against the no-opt-out policy of CPAN are famous.