in reply to Applying the brakes

It's pretty hard to kill most mail-servers just by giving them lots of mail to send. Most reasonable systems (qmail, Ironport, Ecelerity, etc) have solid queuing mechanisms that will deal with any backlog. Just make sure they have enough free disk space and you should be able to push out your messages as fast as you can format them. In general that means using a custom injection format rather than straight SMTP (qmail-inject for qmail, ESMTP for Ironport, Ecelerity::Injector for Ecelerity, for example).

Of course, that assumes the sending server isn't running MS Exchange, or something equally pathetic. If that's the case, by all means throttle it!

Arguably harder than the actual sending is bounce processing. In order to not be blocked by ISPs you have to process bounce logs from your SMTP server and unsubscribe bad addresses. You also have to parse bounces that arrive via email to your Return-Path. Both involve tricky heuristics and very little standardization.

-sam

Replies are listed 'Best First'.
Re^2: Applying the brakes
by grinder (Bishop) on May 08, 2008 at 18:11 UTC

    Oh man, you sinner!

    You talk of reasonable systems, and in the same breath you mention qmail! And you do not speak of Postfix.

    In my mind, the main problem is not going to be the formatting of the messages, nor their queuing on the outbound exchanger.

    To pump out a million or more messages, statistically speaking, all your outbound sockets will quickly be tied up on remote hosts that are out to lunch, overloaded and generally taking too much time to do nothing. This will kill your outbound delivery.

    What you (the OP) needs to do is to set up two machines (or clusters), the first one attempts to deliver the mail at top speed. It performs one try, and one try only, with a very short timeout, 5-10 seconds max. If it fails, it hands the message off to a fallback relay that is configured to be much more lenient and patient at speaking to slow and/or broken exchangers.

    This way your primary outbound exchanger is always available to send out messages, and doesn't get its deferred outbound queue clogged up with messages that might take minutes to deal with. Instead the fallback relay deals with the problem servers.

    It helps if you NAT stuff out through the same IP address, so that greylisting servers don't consider the transaction as new tuple. If you can't then you'll need a third level of fallback relay. The second level makes a couple of attempts to deliver mail on the assumption that the remote server is spitting out transient errors due to greylisting, and only after it gives does it finally divert the message to a third-level fallback relay.

    And naturally, this is simple to configure with Postfix :)

    • another intruder with the mooring in the heart of the Perl

[reply]
      Yeah, I probably shouldn't have included qmail in that list. I've never used it to do 1m+ delivery runs. So far I've only seen Ecelerity and Ironport do that. I included qmail only because I'm pretty sure you couldn't kill it by sending it a ton of mail to send out.

      Neither Ecelerity or Ironport need two machines to deal with slow receivers. You can setup priority levels on different queues if you want to make sure one large mailing can't stop later mails from going out, but the defaults seem to do a pretty good job of that for both, in my experience.

      -sam

[reply]

In Section Seekers of Perl Wisdom