in reply to Sorting a list by frequency of items

This is a sort of non-perl answer, and it may be wholly inapplicable to you.

However, the result your're looking for (minimising nslookups in the MTA) is pretty much only achievable within the MTA itself. I suggest exim http://www.exim.org - it already understands the concept of caching answers and multiple emails destined to the same recipient MTA within the mail queue (and this is smarter than the "domain name" choice, it's based on the MX for that domain)

(p.s. nice name, yojimbo! )

Quick wafting zephyrs vex bold Jim

  • Comment on Re: Sorting a list by frequency of items

Replies are listed 'Best First'.
MTAs
by yojimbo (Monk) on Mar 23, 2001 at 19:49 UTC
    Offishelly, I don't get to choose the MTA, but talking to a couple of other people here, they also send out bulk mails like this to other ISPs so I might be able to set up a dedicated box for this: the server's mail queue is still backed up even after all the work optimising the script :-( If I can sell this idea to management and a few other techs, your recommendation might be very useful. (PS: kewl name yourself :-)
      My vote is for postfix. I've used postfix on several machines. In my experience throughput of a well tuned postfix can be more than 8x what I could get with sendmail on the same hardware. Postfix is also much easier than sendmail to configure (not to mention much more secure). Postfx doesn't have all the features that sendmail does, but it has plenty. I've never needed a feature that postfix didn't have.

      Either way you go I recomend that you throttle your mail generator so it does not overload the mailserver's incoing queue. In my experience MTA performance (postfix, sendmail, or qmail) starts to suffer when the incoming queue gets too large.

        At the web hosting/devopement company I work for, we have a client that has a large mailing list (over 250000, opt-in, yes). When I came on board the situation was that sending the list constantly killed Sendmail and it took nearly six days to send. Some yahoo put a 2 second delay between each email. Of course, they also used perl to make a system call to /var/lib/sendmail. Yuck. That's process/filehandle/cpu hell.

        The solution: P166 with Debain 2.2r2 running Qmail and a change to the way the mail server was used. Instead a making a new connection for each message, one connection (via Net:SMTP) was created and used until the list was done. The sending machine used almost no cpu time (~2-3 %) and hardly any memory (the entire list is never in memory at once.

        The mail machine queue can get as high as 100000, which is insane, but qmail doesn't care. Incoming connections never slow down (that I've noticed, anyway) and surprisingly still seems to actually deliver almost 1 message per second (T1).

        The entire list takes less than two days to finish being delivered. The only problem is that the P166 isn't beefy enough to start qmail-remote processes fast enough. :( (I've set the max at 75, which it hits easily once the incoming onslaught stops and just the queue needs to go out)

        As others have said, it's the job of your MTA to figure out the best way to send the queue. Don't bother with that (IMO) but rather get a POS machine that no one else wants to use and turn it into a bulkmailer.

        gralbe

        gralbe@thebelliconnection.com

        P.S. Feel free to mail me if you have any questions or think I'm an idoit.

      My choice : Sendmail most used, well known, stable ,very portable, and powerful..
      And even if my opinion is biased (Im a sendmail administrator)
      I think sendmail isn't so complicated to configure/manage through the m4 macros.

      Alternative choice : Postfix
      A little bit simpler and said to be more secure and faster
      (even if I'm not aware of any Senmail vulnerability in the current version...)

      In fact it depends on your objective :
      If you want something fast and easy to install to do simple things I think you should use postfix.
      Now if you want to hook some stuffs in your MTA (lots of already developped add-on exists),
      Or if you know you'll do complex things with your MTA, try Sendmail.

      "Trying to be a SMART lamer" (thanx to Merlyn ;-)