in reply to Sending many emails but fast?

Here's some pseudo code describing the algorithm for fast email sending. You could use thread model or fork model for parallel network async processing. I am in favour of the fork model personally.

active_send_mail = 0
threshold = 20

for (each email in the list) {
  send_email(address, content)

  # make sure we don't fork too many processes
  active_send_mail = active_send_mail + 1
  if (active_send_mail >= threshold) {
    waitpid -1
    active_send_mail --
    report on status
  }
}

sub send_email
{
  fork
  if child {
    send_the_email using mail or Net::SMTP
    exit
  }
  # in parent
  return child pid
}

[download]

Replies are listed 'Best First'.
Re^2: Sending many emails but fast?
by Tanktalus (Canon) on Aug 29, 2005 at 14:56 UTC

    Actually, I really doubt this will be fast. In fact, it's likely to become a bottleneck itself. You're chewing up all the CPU's to try to load up the SMTP server with emails, leaving very little left for the SMTP server to use (if it's on the same machine), or leaving very little bandwidth for the SMTP server to send out the emails (if it's on a different machine).

    Better bet is to use the envelope to specify all your recipients and send the mail out once. The SMTP server will only have to process a single message - which it will fork off and send on its own. For example, if 15 recipients are local, they can all be written through immediately without being choked by more incomings. And if another 5 are on another single host, the SMTP server can relay all of those in a single envelope, reducing both bandwidth and CPU time on both machines.

    The only faster way than this is to send each mail to the proper machine directly - and, again, you need to send the mail to each host only once, using the envelope to address to multiple individuals at once on the same host to reduce the number of connections you need to make. But now you're writing your own SMTP server, which is unlikely to be a great use of your time.

[reply]

      Although you are correct that using the envelope to send the same message to multiple recipients on a single mail server is much more efficient, it does not solve the users problem, since it was specifically stated that all messages have dynamic content.

      I would also agree that you should let the mail server do the hard work for you. If you want your program to run quickly, then tell your mailer to queue the message for later delivery. It will return immediately, and be ready for the next message. At some later point (seconds or minutes later), a queue runner will pickup the waiting messages and send them out. This means your program can run quickly, even though the messages will be limited in the speed that your mail server can send them out.

      If you are using qmail, using qmail-inject will do this automatically. If using sendmail, I believe sendmail -ob (or maybe -oq) will do it. If you are sending directly through SMTP, then of course this technique doesn't apply.

[reply]
      Actually, I really doubt this will be fast. In fact, it's likely to become a bottleneck itself. You're chewing up all the CPU's to try to load up the SMTP server with emails, leaving very little left for the SMTP server to use (if it's on the same machine), or leaving very little bandwidth for the SMTP server to send out the emails (if it's on a different machine).
      Presumably you would set threshold to a suitable value to keep the SMTP server going as efficiently as possible. After initially loading up the SMTP server with work to do, the script should spend most of its time waiting.
[reply]
Re^2: Sending many emails but fast?
by themage (Friar) on Aug 30, 2005 at 14:05 UTC
    This looks ok for me as long as you don't use SMTP to send the messages.
    If you do use SMTP, forking and using multiple connection will be slower than use just one connection and eject all the email in the main process.

    I think.
    TheMage
    http://themage.bliker.com
[reply]

In Section Seekers of Perl Wisdom