Using sendmail in large-scale production environments is a dream for IT professionals. Sendmail is free, open source, extremely flexible, and includes commercial support from Sendmail.com if you want it. On the downside, many myths have propagated regarding sendmail's lack of scalability. At O'Reilly's Open Source Convention in Monterey, CA, last month, Nick Christensen attempted to dispel some of those myths.
Nick first pointed out, simply, that sendmail's lack of scalability is in fact a myth. Many organizations with large-scale demands can and do use sendmail, including @Home, AOL, Prodigy, Earthlink, and others. He further pointed out that by far the majority (three out of four) .net domains using mail use sendmail, and that 84 percent of Fortune 100 companies use sendmail in day-to-day operations.
Sendmail obviously can do the job, so what's the trick to scaling it for your network?
Christensen first suggested making mail handling a parallel task. He pointed out that many smaller machines can be significantly cheaper than one big machine and can, in turn, provide better performance as well as fault tolerance.
One way to do this is to set up multiple MX (mail exchanger) records in your DNS, all with equal priority. RFC1123 states that given multiple MX records of equal priority, the sending MTA (mail transfer agent) should randomly pick one of those MX records, which should yield a distributed load. Not all MTA's do this, however, so this method may not yield the statistically balanced pattern you want.
Another possibility is to use what is known as round-robin DNS. This feature, available in most current name servers (including any version of BIND greater than 4.9.2) allows you to assign two IP addresses to a single hostname, which will then be returned in rotating order. For example, if you have something like: a.mx.hugeisp.net. IN A 10.3.1.11a.mx.hugeisp.net. IN A 10.3.1.12The first time you query for a.mx.hugeisp.net, you'll get 10.3.1.11 and 10.3.1.12 in that order. The next time the DNS server answers that query, it will put 10.3.1.12 first. Since the DNS client (in this case, the sending MTA) will usually use the first IP address it receives, this will yield balanced traffic patterns in almost all cases.
The benefit to doing it this way is that it will almost always yield the traffic pattern you want, and it allows you more fine-grained control of mail handling. Christensen used the following example to illustrate this: hugeisp.net. IN MX 10 a.mx.hugeisp.net.
IN MX 10 b.mx.hugeisp.net.
a.mx.hugeisp.net. IN A 10.3.1.11
a.mx.hugeisp.net. IN A 10.3.1.12
b.mx.hugeisp.net. IN A 10.3.1.21
b.mx.hugeisp.net. IN A 10.3.1.22
Now, you could have four email servers sharing the load, but if you take advantage of both RFC1123 "rotation" as well as DNS round-robin rotation, it should yield a fairly balanced distribution of traffic.
Christensen then showed us the DNS settings of several ISPs, with attention to the multiple MX records and multiple A records used to handle particular settings:
9 MX records of precedence 15
5 A records for each MX record
45 hosts total handling incoming email
Only 15 A records returned
No lower precedence MX records
1 MX record for precedences 5,10,15,30
3 A records for precedences 5,10,15 mail-intake-.mail.com 2 A records for precedence 30 spool.mail.com Hierarchical with failover to lower precedence servers @Home 1 precedence 50 MX record with 10 A records (mx-rr.home.com) 8 precedence 100 MX records, some same as above (mx[1-4]-[ew].mail.home.com 2 precedence 150 MX records, mx.home.com There are no A records for home.com Outgoing mailThat's all well and good for inbound mail, but how do you handle large amounts of user-generated outbound mail? Again, Christensen says parallel machines are the way to go.
Have all machines that generate email send it to smtp-out.hugeisp.net (Christensen's example). If the machine in question is using sendmail to originate the mail, you can place the line DSsmtp-out.hugeisp.netin its config file. This tells the machine to forward all its mail to a smart relay. Sendmail looks for an MX record for this address first, before it looks for an A record, so you could then put the following in your DNS zonefiles: smtp-out.hugeisp.net. IN MX 10 a.out.hugeisp.net IN MX 10 b.out.hugeisp.netIf your network includes desktop machines that aren't using sendmail (and that don't really contain a decent MTA), you can set up one machine to accept their mail and immediately relay it out via MX records to other machines for them to handle.
Another often forgotten feature of sendmail is the fallback MX host. This is used when outgoing mail can't be delivered to any of the usual MX hosts. If you configure a machine to be the fallback MX host for your outgoing mail, you can move "problem mail" for sites that are down or unresponsive to a different physical machine, possibly using radically different queuing properties. This might be as simple as an increased delay between queue retries, or it might be something like holding onto the mail far longer than normal before bouncing it as undeliverable. This allows your main outbound mail server to concentrate on moving mail, eliminating worries that problem mail will clog up the queue. All the problem mail then ends up in one place, to be delivered at the leisure of the other server, without affecting the main server's performance.
As you scale your mail installation, it's important to create a low-priority offsite "spillover" host. This machine must be housed somewhere other than where your main mail servers are (preferably on a different network completely). It should have a very low MX priority and be used only as a last resort for people on the outside world trying to send mail to you. In the case of failure within your mail network, this host would get your inbound mail off the Net and onto your servers. You can configure the spillover host to queue mail for the main mail servers for as long as your needs warrant, preventing you from losing mail in the unlikely event of a long-term outage (such as those displayed by recent MCI/WorldCom frame-relay problems).
At the end of the day, maintaining a working, scaled sendmail installation is actually less about software patches and obscure configuration settings and more about knowing how mail works and taking advantage of everything mail RFCs and BIND DNS software have to offer you.
Will sendmail 8.10 help you even more?
At the recent O'Reilly Open Source Convention in Monterey, CA, Greg Shapiro unveiled a feature list for what's coming in sendmail 8.10. Here's a sneak peek at a few of these new features:
Multiple queues on the same machine
This feature could partly eliminate the need for the fallback MX host as a means to improve scaling, since it allows you to create multiple queues with unique properties (for delivery retry frequency and so on) on the same machine. Because the different queues are on the same machine, however, they do have the potential to fill up disk space quickly in the event of the failure of a high-traffic site, so the fallback MX host won't go away anytime soon.
Separate directories for qf and df files This allows for faster access to files in large spool directories, because the number of file entries per directory will be lower. It also allows the oft-accessed qf files to be stored on high-speed solid-state media, while the less-active df files can be stored on conventional storage media.
Improved LDAP capability
LDAP use in 8.10 has been improved. It now uses asynchronous lookups to save resources as well as caching connections to the LDAP server so that multiple maps can share a single connection.
Buffered file I/O
This new feature improves performance and grants the ability to configure timeouts on DNS queries. Of course, one other mandatory scaling feature going forward is IPv6 support, and sendmail 8.10 is quite capable of dealing with that as well.
Sendmail has continued to improve over time, and 8.10 shows no signs of breaking that tradition.