A large portion of the delay when sending mail is the time spent communicating with the server. If you are sending identical content to these recipients then you could preprocess the address list and batch them by domain, with a max of 5ish people in a batch - to avoid cases where a large delivery list is treated as spam by the receiving server.

You could also look at reusing the MIME::Lite object you created, changing the recipient and sending again. I haven't check the documents to see if this is supported, but it is worth researching to reduce time spent constructing the object.