Indeed, the internals to MIME::Lite will die if the RECIPIENT command fails when sending via SMTP, such as if the e-mail address is invalid (see the latest source in the send_by_smtp method). The easiest way around this is to wrap the send call inside an eval.
Barring that, you'll have to use something else other than MIME::Lite's send, such as utilizing another module.