I guess that's a common problem, and some mail servers already have a solution for that.

Perhaps you can query the local queue size of your mail server and sleep if there are more than $limit mails in the queue.

Especially when it comes to concurrency I try very hard to find some piece of software that already solves the hard part of the problem, in this case I'd hope for a decent mail server.

First, it refuses connections and new RCPTs when it's already sweating. Second, I try not to get it to that point. (This send_mail function is from my own Net::SMTP::OneLiner.)

# This part is a no brainer and your mail software
#   likely already supports something like it:

eval { send_mail('postmaster@mei.net', "$dir_entry\@mei.net", 
    "Usage Notice (full mailbox)", $msg) };
if( $@ ) { 
        warn "sleeping for 2 seconds then retrying due to send_mail() 
+WARNING: $@";
        sleep 2;
        redo RETRY;
    }
}
[download]

The second thing is to simply wait until the load average is low enough:

sub sleep_until_low_load {
    my $limit = shift;

    REDO:
    open PROC, "/proc/loadavg" or die $!;
    my $line = <PROC>;
    my $load = $1 if $line =~ m/^\s*([\d\.]+)\s+/;
    close PROC;

    if( $load > $limit ) {
        print "\tsleeping for 5 seconds since $load > $limit\n";
        sleep 5;
        goto REDO;
    }
}
[download]

open my $exim, "| exim -bS" or die;

for (...) {
  print $exim <<'MSG';
MAIL FROM: foo@bar.com
RCPT TO: doz@bar.com
DATA
Subject: Your Invoice
To: You
From: Me

You have to pay 1,000,000 dollars
.
MSG

}
[download]

my $workers = 4;
for my $ix (0..$workers-1) {
  unless (fork) {
    open my $fh, '<', $address_list_file_name or die;
    open my $exim, "| exim -bS" or die;
    while (<>) {
      next unless (($. % $workers) == $ix);
      send_invoice($exim, $_);
    }
    exit 0;
  }
}
[download]

You may find this comparison of MTAs interesting. Yes it is dated but Exim was slow as a dog back then. Here is another comparison which is a bit less dated and still shows Exim at the back of the field.

Now the hardware used was relatively old and slow but this is not a hardware bottleneck task. It is concurrency, network connectivity, and DNS lookups that are bottlenecks. To send an email your server has to look up the address in DNS (you want a big local cache), then make the connection to the remote server (may take seconds), then send HELO,MAIL,RCPT,DATA,EOM - all of which can be slowed down by the remote server. The entire transaction can easily exceed several seconds, thus you require multiple concurrent processes (50+) in your MTA to get any sort of decent throughput.

You will note that performace in terms of emails per second is a feeble 5-15 depending on MTA. At 20/second througput you are looking at 20 hours runtime for 1.5 million emails.

I suggest you look at a dedicated mail server that also runs its own DNS with a huge cache to do this task. Exim looks like one of the less efficient options judging from the performance benchmarks. If you dump 1.5 million emails onto your Exim queue and it is really only running a throughput of 5/sec you will effectively freeze outgoing email for 80 hours as any new message will go to the back of the queue. Even if it is doing 50/sec there will be nothing else going out for 8 hours - unless of course there is some way to flag it as low priority. Not only will you have an outgoing email problem you will also have an incoming email issue as Exim is also recieving incoming connections.

If you did want to split up the flatfile of email addresses you don't need a database. Just use split(1).

good info, thanks for this.

Just had another thought. While dumping the file (probably in paced chunks - waiting for the queue to shrink back towards zero) may be sensible you need to permute you infile somewhow to ensure you don't have:

bob@domain
sue@domain
...
foo@domain
bar@other_domain
[download]

If you dump a whole series of emails to the same mail server in a row it will choke and possibly ban/throttle you. One simple approach would be simply to apply a sort and let the variation in username vaguely randomise the domains or you could shuffle them in an array using a Fisher Yeats.

Provided you don't have high frequencies of gmail, hotmail, yahoo accounts a simple sort ought to work OK, otherwise you may need some clever code to make sure that these common domains don't occur in a row.

I would probably take the easy road and try a simple sort first and check how many times a given domain occurs in your proposed concurrency frame (probably 50-100). Domains occuring more than 2-3 times within a frame may be a problem as your MTA will be asking for that many concurrent connections.

Update

Could not resist. Here is a don't hit the same domain if we have sent an email in the last n width frame algorithm to run you address list through. NB Code updated to remove bug where domain pulled off fifo in else unchecked against current working domain - if it is that needs to go on the fifo, if not it is good to go (untested)

Read more... (2 kB)

Even if your mail-server can handle a few hundred messages a second it will still take many hours to process your batch of messages and it seems unlikely to me that preparing these messages by your script will take that long. Do you have any idea how long it takes to prepare your messages? As long as it is faster than the sending of the messages there is no need to complicate matters to start additional threads or use forking.

The idea behind forking or threading is to offload the queue i'm processing to the mail server asap. what i want is a process that runs in as short a time as possible.

for example, if someone has to kick of the notification process over the weekend, it would really suck to have to come in/log in and check it every few hours to make sure its still going.

..and yes, i'm aware of writing automated monitors and all that, but at the end of the day, nothing beats (at least initially in the first few runs) eyes-on to ensure things are going smoothly.

if you really really want communication, then the database all the hosts are already accessing gives a convenient means to coordinate.
For example it could give out contingents of email adresses to each host that are put in by one coordinator process at the specified rate.

Perl threads could be used similary, one coordinator thread and many sub-threads. Counter question: Safe from what?

Of course, that assumes the sending server isn't running MS Exchange, or something equally pathetic. If that's the case, by all means throttle it!

Arguably harder than the actual sending is bounce processing. In order to not be blocked by ISPs you have to process bounce logs from your SMTP server and unsubscribe bad addresses. You also have to parse bounces that arrive via email to your Return-Path. Both involve tricky heuristics and very little standardization.

-sam

Oh man, you sinner!

You talk of reasonable systems, and in the same breath you mention qmail! And you do not speak of Postfix.

In my mind, the main problem is not going to be the formatting of the messages, nor their queuing on the outbound exchanger.

To pump out a million or more messages, statistically speaking, all your outbound sockets will quickly be tied up on remote hosts that are out to lunch, overloaded and generally taking too much time to do nothing. This will kill your outbound delivery.

What you (the OP) needs to do is to set up two machines (or clusters), the first one attempts to deliver the mail at top speed. It performs one try, and one try only, with a very short timeout, 5-10 seconds max. If it fails, it hands the message off to a fallback relay that is configured to be much more lenient and patient at speaking to slow and/or broken exchangers.

This way your primary outbound exchanger is always available to send out messages, and doesn't get its deferred outbound queue clogged up with messages that might take minutes to deal with. Instead the fallback relay deals with the problem servers.

It helps if you NAT stuff out through the same IP address, so that greylisting servers don't consider the transaction as new tuple. If you can't then you'll need a third level of fallback relay. The second level makes a couple of attempts to deliver mail on the assumption that the remote server is spitting out transient errors due to greylisting, and only after it gives does it finally divert the message to a third-level fallback relay.

And naturally, this is simple to configure with Postfix :)

• another intruder with the mooring in the heart of the Perl

Yeah, I probably shouldn't have included qmail in that list. I've never used it to do 1m+ delivery runs. So far I've only seen Ecelerity and Ironport do that. I included qmail only because I'm pretty sure you couldn't kill it by sending it a ton of mail to send out.

Neither Ecelerity or Ironport need two machines to deal with slow receivers. You can setup priority levels on different queues if you want to make sure one large mailing can't stop later mails from going out, but the defaults seem to do a pretty good job of that for both, in my experience.

-sam

It sounds to me like your admins are right. You see, even a single thread can “outrun the mail-server.” Therefore, there's no advantage to having more than one thread or process doing that, plus it's considerably simpler to manage.

If the mail-server is such that it can simply be given an enormous queue of stuff to chomp-through, then just give it the enormous queue all at once and let it chomp-through as it will. If there's an issue of choking-out other mail that needs to go out the same tube, any sort of “simple throttle” should do ... in one process ('cuz you just don't need more).