in reply to Fast dispatch of e-mails

szabgab:

Everyone here has already made good points. My production strategy is a combination of clint's earlier post, and the comments made tirwhan.

Background: I always want to return my webrequest as soon as possible. Sometimes the email I need to trigger out have 500k or 5MB attachments. I don't want two strategies -- one that works for really tiny emails and one that works for huge attachment emails. So I found the following to be an excellent compromise for my setup.

Solution: Anytime my webapp needs to send out an email, a smaller descriptor email goes out to 'processemail@localhost', which is injected almost immediately. The injected descriptor email ultimately gets passed along to another process that sends out the describe email, for the webapp. The descriptor email is arbitrary and generally a tiny protocol that you create. The protocol messages shouldn't contain the attachments, but rather refer to WHERE the attachments can be attached. You could call this description email a meta-email: an email about an email (or just flame me for using the word meta instead :-)) This is generally very fast and pretty scalable.

Once the meta-email arrives to the 'processemail@localhost' address (it didn't have to travel far) a script POSTs the email's content to a local mod_perl service at 'http://localhost/processemail'. This services creates a MIME::Lite object, assembles it from the meta-email body (as a POST value), attaches the attachments from the references, and injects the 5MB monster into a local MTA (which can take a second or two.) My local smtpd serer (the one on the webserver) then does smart forwarding to our real outbound mailer. If our outbound mailer is down, it's sits on the webservers smtpd queue as necessary. If the the webservers smtpd is down, um... we're screwed ;-)


This strategy can rule the day if you have a robust MTA, and don't mind the size/network penalty for the descriptor emails. This strategy can also be very flexible. For example, in the descriptor email your attachments could be described as URLs which the 'processemail' server retrieves and attaches. This could then allow you to move the MTA from where the attachments are stored.

It scales pretty well too. For example, each webserver has it's own mail queue which means its output can be throttled to slow delivery to the outbound mailer, but still return immediately to webrequests.

Anyhow, just some more food for thought. Have a great weekend!

Kurt