althepal has asked for the wisdom of the Perl Monks concerning the following question:

I've written my own SMTP server in perl. I know there are a lot out there already, but I wanted to write my own, OK? (defensive mode off now).

Over time, I've been adding functionality (i.e. modules) and the server footprint is getting big (approx 15-16Mb of memory). Mail::SpamAssassin adds about 9Mb to the footprint, IO::Socket::SSL adds a few meg, etc.

When each request comes in, I fork a new server to handle it. When I run the top command, I see that nearly all the memory used by the forked processes is Shared (i.e. 15Mb out of 16Mb). I take this to mean that the overhead in forking a new process is fairly low.

My question is, is there anything fundimentally wrong with having a server with such a large footprint? I notice that most servers I see are considerably smaller (e.g. httpd and sendmail are 3-4Mb). I am assuming that loading everything at the beginning is the most efficient way of handling the server, but I wonder if I'd be better off loading some modules as needed (like SSL) or running a separate server for some (like SpamAssassin).

I want this to be a high performance server. It currently handles on the order of 50,000 messages a day, and the hardware it's on is mainly sitting idle. I don't really see any performance bottlenecks yet, but I'm thinking about the future. It's hard to test different configurations, because everything goes so fast, I can't really notice any differences.

Everything is working great, I'm just wondering if I'm approaching this correctly. Any feedback is appreciated.

- Alex Hart

Replies are listed 'Best First'.
Re: SMTP server in perl
by Joost (Canon) on Apr 07, 2005 at 22:11 UTC
    If you're forking, load as much code and unchanging data as soon as possible, since it should all be shared.

    By the way, 16 Mb is not that much if you're sharing most of it; I've seen apache/mod_perl applications that take more than 150 Mb (almost all of it shared) and still have very good performance.

    Say you need 1Mb unshared per process, + 15 Mb for the whole set, on a machine with 500 Mb memory, that would mean you could run at least 400 concurrent processes; I would be surprised if you would need so many with 50.000 messages a day - but ofcourse that depends on the amount of time you need to spend on a single message.

Re: SMTP server in perl
by samtregar (Abbot) on Apr 07, 2005 at 23:06 UTC
    I want this to be a high performance server. It currently handles on the order of 50,000 messages a day, and the hardware it's on is mainly sitting idle. I don't really see any performance bottlenecks yet, but I'm thinking about the future. It's hard to test different configurations, because everything goes so fast, I can't really notice any differences.

    If you're serious about wanting to make your server faster then you're going to have to build a benchmarking framework. This should consist of a client (possibly multiple clients) which fire messages at the server at a configurable rate. The benchmarking system should record the time needed for each message to be accepted by the server as well as the time needed for delivery (if you server does delivery).

    Here's an example of the kind of report you should try to generate: http://www-dt.e-technik.uni-dortmund.de/~ma/postfix/bench2.html

    Once you have a benchmarking system you'll be able to make a small change (like dynamically loading SSL on demand) and test to see how that effects performance. Rinse, repeat and eventually rewrite the whole thing in C!

    -sam

    PS: To answer your actual question - 16MB is rather high for an SMTP server but whether its too high depends entirely on your needs. If your system can handle dedicating 16MB plus around 1MB per connection to handling incoming email then there's no reason for you to try to make it lower.

Re: SMTP server in perl
by sh1tn (Priest) on Apr 07, 2005 at 22:16 UTC
    As fas as performance is a concern you may want to consider
    non forking and non blocking behaviour.


Re: SMTP server in perl
by Eyck (Priest) on Apr 08, 2005 at 07:42 UTC
    Take a look at postal or rabid, http://www.coker.com.au/postal/, mail benchmarking tools.

    You seem to be thinking that performance is non-issue, because your hardware is able to handle your current load, but remember that on internet everything is spiky, it's enough for one spammer, or few bots to target your server and you're in big trouble with your 16M processes trying to fork fast enough to handle incoming connections...

    Having said that, I have succesfully deployed perl-based servers, that were able to handle such situations, the key for pre-forking server is to keep large enough spool of idle threads.

    And the best trick is to limit amount of incoming connection at the firewall - say you found through your benchmarks that at 30 requests/second your machine starts melting, just set your firewall to accept SYN packets to port 25 with rate less then 25.

    That way under normal circumstances you'll be allright, and during spike/scan or attack you'll still be handling your users.