It can be made less annoying if you just use the challange/responce once to verify each new email address that send to the list. Reply back with a message that has a encoded url that verifies the email address as "ok" and also maybe have a way to reply to the message as an ok trigger. to quadruple the number of messages you are sending would be impossible. Worst case you would send two times as many messages assuming that there is never a repeat poster on the list.
If that process puts too much load on your email server there is something wrong, There is no way that method could be more intensive than SpamAssasin or some other filter software.