PRINCIPLES – DNS – ARCHITECTURES – SPAM Electronic Mail PRINCIPLES – DNS – ARCHITECTURES – SPAM Dagens föreläsning handlar om katalogtjänster.

Mail system components SMTP MDA (Local) MTA MDA (SMTP) MDA (SMTP) MTA MDA (Local) MSA MSA SMTP SMTP IMAP/POP3 IMAP/POP3 MAA MUA MUA MAA It’s a mess. Really. M stands for Mail, A for agent. MDA MTA MSA MAA MUA MRA are standard components. Not all components may be separate pieces of software, and not all need be present at all. MRA MUA

MUA Mail User Agent Reads and writes e-mail Writes e-mail to MTA using SMTP (usually) Reads e-mail delivered by MDA or retrieved by MRA Examples Mozilla Thunderbird Outlook Express

MRA Mail Retrieval Agent Retrieves e-mail from MAA Makes mail available to MUA Examples Fetchmail Integrated in Thunderbird etc. MRA is not always used, but sometimes. It’s sort of ”pull” for mail.

MAA Mail Access Agent Authenticates MUA/user Reads e-mail from mailbox Makes e-mail available to MUA Examples Courier IMAPD

MSA Mail Submission Agent Accepts mail from MUA Prepares and delivers to MTA Examples Postfix postdrop+pickup sendmail-msa

MTA Mail Transfer Agent Routes incoming mail Determines which MDA to send mail to May alter mail being routed Examples Postfix cleanup + qmgr + + trivial-rewrite Sendmail

MDA Mail Delivery Agent Accepts mail from MTA Delivers mail to local mailbox Delivers mail to other MTA Examples Postfix local, smtp, pipe mail.local (local delivery) procmail (local delivery)

Mail system components SMTP MDA (Local) MTA MDA (SMTP) MDA (SMTP) MTA MDA (Local) MSA MSA SMTP SMTP IMAP/POP3 IMAP/POP3 MAA MUA MUA MAA It’s a mess. Really. M stands for Mail, A for agent. MDA MTA MSA MAA MUA MRA are standard components. Not all components may be separate pieces of software, and not all need be present at all. MRA MUA

Path of a message 1. New message written Mail is written in MUA Mail is submitted to MSA 2. Reception of message at MSA Authenticate the user Is user authorized to send mail? Rewrite headers if needed Submit message to MTA Report success to MUA 3. Reception of message at MTA Is sender valid and permitted Is recipient valid and permitted Are contents valid and permitted Run applicable content filters Rewrite header Select appropriate MDA Submit mail to MDA If unsuccessful, queue mail for later delivery

Path of a message 4. Reception of message at MDA Send message to remote SMTP server (example) 5. Rcpt of message at remote MTA Same as step 3 Submit message to MDA 6. Reception of message at MDA Check applicable filters/rules Deliver message to local mail store 7. MAA detects new message Signal new message to MUA 8. Message read in MUA Retrieve message from MAA Present message to user

Mail system components SMTP MDA (Local) MTA MDA (SMTP) MDA (SMTP) MTA MDA (Local) MSA MSA 3. Reception of message at MTA Is sender valid and permitted Is recipient valid and permitted Are contents valid and permitted Run applicable content filters Rewrite header Select appropriate MDA Submit mail to MDA If unsuccessful, queue mail for later delivery 5. Rcpt of message at remote MTA Is sender valid and permitted Is recipient valid and permitted Are contents valid and permitted Run applicable content filters Rewrite header Select appropriate MDA Submit mail to MDA SMTP SMTP IMAP/POP3 IMAP/POP3 MAA MUA MUA MAA It’s a mess. Really. M stands for Mail, A for agent. MDA MTA MSA MAA MUA MRA are standard components. Not all components may be separate pieces of software, and not all need be present at all. 7. MAA detects new message Signal new message to MUA 8. Message read in MUA Retrieve message from MAA Present message to user 6. Reception of message at MDA Check applicable filters/rules Deliver message to local mail store 4. Reception of message at MDA Send message to remote SMTP server (example) 2. Reception of message at MSA Authenticate the user Is user authorized to send mail? Rewrite headers if needed Submit message to MTA Report success to MUA 1. New message written Mail is written in MUA Mail is submitted to MSA MRA MUA

Postfix architecture trivial-rewrite trivial-rewrite local Incoming queue Active queue local  File, Command Network  qmqpd lmtp  Network cleanup qmgr Network  smtpd smtp  Network pickup virtual  File pipe  Command Postdrop queue postdrop Deferred queue Local  sendmail

SMTP Simple Mail Transfer Protocol Default: TCP port 25 RFC 2821 ESMTP – Extensions Text-based protocol No encryption No authentication Example %telnet mail.ida.liu.se 25 Trying 130.236.177.25... Connected to mail.ida.liu.se. Escape character is '^]'. 220 mail.ida.liu.se ESMTP Sendmail 8.13.3/8.13.3; Mon, 29 Aug 2005 13:34:42 +0200 (CEST) EHLO lap-65.ida.liu.se 250-mail.ida.liu.se Hello gedrix.ida.liu.se [130.236.176.11], pleased to meet you MAIL FROM:<davby@ida.liu.se> 250 2.1.0 <davby@ida.liu.se>... Sender ok RCPT TO:<info@samizdat.se> 250 2.1.5 <info@samizdat.se>... Recipient ok DATA 354 Enter mail, end with "." on a line by itself From: David Byers <davby@ida.liu.se> To: Samizdat Info <info@samizdat.se> Date: Mon, 29 Aug 2005 13:38:00 +0100 Subject: Testing SMTP Body text. . 250 2.0.0 j7TBYgXf001051 Message accepted for delivery QUIT 221 2.0.0 portofix.ida.liu.se closing connection

The role of DNS Problem: which server to contact for a mail address? Solution: look up MX record in DNS Example: example.com MX 10 mail1.example.com. example.com MX 10 mail2.example.com. example.com MX 20 mx.nodomain.edu. Explain what MX records are good for. Talk about SRV records as a generalization of the same. Talk about what happens if there is no MX record. Note that MX records are valid for the name only, not subdomains. Explain the priority field.

Structure of a message Envelope From SMTP dialog Header Message metadata Specified in RFC 2822 Body Sequence of ASCII characters Separated from body by CRLF Syntax message::= headers CRLF body headers ::= header headers | e header ::= name ”:” value CRLF Return-Path: andjo@ida.liu.se Delivery-Date: Fri May 14 08:38:34 2004 Received: from diag7.ida.liu.se (diag7.ida.liu.se [130.236.177.217]) by portofix.ida.liu.se (8.12.11/8.12.11) with ESMTP id i423760; Fri, 14 May 2004 08:38:31 +0200 (MEST) Received: (from andjo@localhost) by diag7.ida.liu.se (8.12.10+Sun/8.12.10/Submit) id i401197; Fri, 14 May 2004 08:38:31 +0200 (CEST) Date: Fri, 14 May 2004 08:38:31 +0200 From: Andreas Johansson <andjo@ida.liu.se> To: David Byers <davby@ida.liu.se> Cc: pjn@ida.liu.se Subject: Re: =?ISO-8859-1?Q?N=E4tverksproblem?= Message-Id: <20040514083831.495c66a2.andjo@ida.liu.se> In-Reply-To: <41r7togybd.fsf@obel19.ida.liu.se> References: <41r7togybd.fsf@obel19.ida.liu.se> Organization: =?ISO-8859-1?Q?Link=F6pings?= universitet X-Mailer: Sylpheed version 0.9.6 (GTK+ 1.2.10; sparc-sun-solaris2.9) Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 X-Virus-Scanned: clamd / ClamAV 0.70, clamav-milter 0.70j X-Spam-Flag: NO X-Scanned-By: milter-spamc/0.15.245 ( [130.236.177.25]); pass Walk through the important headers: Received Note header rewriting has taken place Note that from line means nothing Note X headers

Some things the MTA does Header rewriting Addition of received header Addition of Message-ID Address rewriting Example rewriting Removal of host name Addition of domain name davby  david.byers Mail routing Find server for recipient address Queue mail for submission Re-submit failed messages Send delivery status notifications SMTP example Find MX records for address Try records in ascending order Try A record for address

Postfix address rewriting Rewrite addresses to standard form Canonical address mapping Address masquerading Automatic BCC recipients Virtual aliasing Resolve address to destination Mail transport switch Relocated users table Generic mapping table Local alias database Local per-user .forward files Local catch-all address smtpd qmqpd pickup cleanup trivial-rewrite (rewrite) Incoming queue Deferred queue Active queue qmgr trivial-rewrite (resolve) smtpd lmtp local

Postfix address rewriting Rewrite addresses to standard form Canonical address mapping Address masquerading Automatic BCC recipients Virtual aliasing Resolve address to destination Mail transport switch Relocated users table Generic mapping table Local alias database Local per-user .forward files Local catch-all address smtpd qmqpd pickup Examples user%domain -> user@domain user -> user@myorigin user@host -> user@host.mydomain cleanup trivial-rewrite (rewrite) Incoming queue Deferred queue Active queue qmgr trivial-rewrite (resolve) smtpd lmtp local

Postfix address rewriting Rewrite addresses to standard form Canonical address mapping Address masquerading Automatic BCC recipients Virtual aliasing Resolve address to destination Mail transport switch Relocated users table Generic mapping table Local alias database Local per-user .forward files Local catch-all address smtpd qmqpd pickup cleanup trivial-rewrite (rewrite) Examples user -> first.last user@host.mydomain -> user@mydomain user@dom -> user.dom@mydomain first.last -> user Incoming queue Deferred queue Active queue qmgr trivial-rewrite (resolve) smtpd lmtp local

Postfix address rewriting Rewrite addresses to standard form Canonical address mapping Address masquerading Automatic BCC recipients Virtual aliasing Resolve address to destination Mail transport switch Relocated users table Generic mapping table Local alias database Local per-user .forward files Local catch-all address smtpd qmqpd pickup cleanup trivial-rewrite (rewrite) Incoming queue Deferred queue Active queue Examples user@site -> newuser@newsite qmgr trivial-rewrite (resolve) smtpd lmtp local

Postfix address rewriting Rewrite addresses to standard form Canonical address mapping Address masquerading Automatic BCC recipients Virtual aliasing Resolve address to destination Mail transport switch Relocated users table Generic mapping table Local alias database Local per-user .forward files Local catch-all address smtpd qmqpd pickup cleanup trivial-rewrite (rewrite) Incoming queue Deferred queue Active queue qmgr trivial-rewrite (resolve) Examples user@localdomain.local -> newuser@domain smtpd lmtp local

Postfix address rewriting Rewrite addresses to standard form Canonical address mapping Address masquerading Automatic BCC recipients Virtual aliasing Resolve address to destination Mail transport switch Relocated users table Generic mapping table Local alias database Local per-user .forward files Local catch-all address smtpd qmqpd pickup cleanup trivial-rewrite (rewrite) Incoming queue Deferred queue Active queue qmgr trivial-rewrite (resolve) smtpd lmtp local Examples root -> user first.last -> user

More things the MTA does Content filtering Scan for virus content Check validity of attachments Check size of messages Check for spam Common spam checks Keyword/feature filters IP address blacklists Greylisting Authorization Check submitting host IP Check from address Check recipient address Check contents Typically permitted Mail from local systems Mail to local recipients Mail from trusted hosts to any recipient

Postfix content filters Three flavors Built-in content inspection After queue content filter Before queue content filter ”Body checks”, ”Header checks” Regexp checks on message For stopping specific problems pickup qmqpd smtpd (filter) cleanup bounce local (forwarded) Incoming queue Postfix has three types of content filters. Built in content inspection is known as header and body checks, and are regular expressions that are matched against the message. These are to stop specific problems, not for general spam filtering. Since this filtering is done on-line, if it takes too much time, the cleanup process will take up all CPU time, and the number of simultaneous SMTP connections will reach the maximum.

Postfix content filters Three flavors Built-in content inspection After queue content filter Before queue content filter After mail has been accepted Heavy-duty filtering Need to avoid spurious DSNs pickup qmqpd smtpd (filter) cleanup bounce local (forwarded) Postfix postdrop Postfix queues Postfix has three types of content filters. After queue content filters are run after messages are accepted for delivery. This allows the SMTP dialog to finish before the content filter runs. In systems that accept large amounts of mail, this may be necessary to avoid running out of available sockets. However, after queue content filters have no reliable way of notifying the senter of problems. Postfix sendmail qmgr Content filter pipe smtp local

Postfix content filters Three flavors Built-in content inspection After queue content filter Before queue content filter Before mail is accepted Heavy-duty filtering Only for low-volume sites smtpd Content filter smtpd (filter) cleanup Postfix has three types of content filters. Before-queue content filters run while the SMTP dialog is active, so the content filter can reject the message as part of the SMTP dialog. The obvious problem is that the content filter increases SMTP response time. Under high load, the delay may be sufficient to cause the SMTP client to time out. Incoming queue

Spam protection Relay restrictions Greylisting DNS-based blocklists

SMTP Restrictions Denying relay Greylisting in action 220 solzhenitsyn.samizdat.se ESMTP Postfix HELO gedrix.ida.liu.se 250 solzhenitsyn.samizdat.se MAIL FROM:<davby@ida.liu.se> 250 Ok RCPT TO:<byers@lysator.liu.se> 554 <byers@lysator.liu.se>: Relay access denied Greylisting in action 220 portofix.ida.liu.se ESMTP Sendmail 8.13.3/8.13.3 HELO metzengerstein.visit.se 250 portofix.ida.liu.se Hello metzengerstein.visit.se [212.214.112.221] MAIL FROM:<info@samizdat.se> 250 2.1.0 <info@samizdat.se>... Sender ok RCPT TO:<davby@ida.liu.se> 451 4.7.1 Greylisting in action, please come back in 00:10:00 SMTP restrictions are restrictions implemented by the SMTP server. Typically they look at the headers, the network connection and the SMTP dialog only, and are very lightweight. For example, a common restriction is to refuse mail from external IP addresses that is not addressed to local addresses.

DNS-based blocklists Blocklists work like in-addr.arpa lookup %host -t TXT 1.185.167.195.l1.spews.dnsbl.sorbs.net 1.185.167.195.l1.spews.dnsbl.sorbs.net TXT "! [1] XPD Limited, see http://spews.org/ask.cgi?S3007" %host -t TXT 1.185.167.195.dnsbl.sorbs.net 1.185.167.195.dnsbl.sorbs.net TXT "Spam Received See: http://www.sorbs.net/lookup.shtml?195.167.185.1" Talk about how they work. Talk about the different policies of blocklists and that this means you have to choose carefully which ones to actually use.

Architectures

Client-side outgoing mail Client has an MTA Client MTA queues mail Client has no MTA If central MTA is down, no mail can be sent Client MTA MUA Client MUA The part of the e-mail system closes to the user is the MUA. Here there are a number of options. Traditionally, unix systems have an MTA on each client. This allows the client to queue outgoing mail regardless of whether the destination is ready to accept it or not. Having an MTA on the client increases complexity somewhat, but also increases robustness. Often, the integrated MTA will not be an SMTP server. Today, most users do not have a local MTA. Instead, they use MUAs that connect directly to an external MTA. This way, if the central MTA is unavailable, no mail can be sent. On the other hand, users will get immediate feedback on any problems in the system. MTA Client MUA

Outgoing mail Smarthost All mail through one MTA Central configuration Failover Better reliability Load balancing Better performance Client Client MTA Client Client MTA Client MTA Client Most architectures are a variant of a smarthost architecture. Here, all main from within a network is transferred to a dedicated mail server, called the smarthost, which is responsible for things like address rewriting, filtering and so forth. The MTAs on each individual client may be as simple as a program that can communicate with the smarthost using SMTP – there is no need for an SMTP server on each client (although there might be one). This architecture permits centralized administration and very simple client configuration. However, the basic smarthost system has a SPOF in the smarthost. To improve reliability, failover needs to be employed. There are several ways to accomplish this, but the most reliable method is to set up two servers that monitor each other. When one goes down, the other takes over its IP address and possibly MAC address. Failover doesn’t address scalability very well. A layer 7 switch, or better still a pair of them, can balance the load from all clients over all MTAs. The layer 7 switches can also monitor the MTAs to avoid those that are offline. Client Client Client Client L4 Switch MTA Client MTA Client L4 Switch MTA Client Client Client

One front-end multiple backends Incoming mail Multiple MTAs MTA MDA MTA MDA Load-balanced MTAs MTA MDA MTA MDA L4 Switch MTA MDA L4 Switch MTA MDA One front-end multiple backends For incoming mail, there are several concerns: the load must be managed; the service must be reliable; configuration must be sensible; and so on. Organizations with lots of mail will want to have multiple MTAs. This can easily be accomplished by publishing multiple MX records with equal preference in DNS. It can also be accomplished using an L7 switch. An L7 switch will guaranteee even load balancing, which DNS tricks won’t. On the other hand, an L7 switch is an additional complication. In some systems, a hierarchy of MTAs will be used. The top-level MTA transfers mail to MTAs deeper in the hierarchy. This is generally useful for an organization that wants one public-facing MTA, but in reality has lots of internal systems. How mail is distributed between internal MTAs depends on the situation. One possibility is to have multiple subdomains. Another is to have the public-facing MTA rewrite addresses into multipls subdomains. There are other ways too. In these architectures, we have multiple MTAs, but all mail generally needs to go in a single mail storage system. Luckily, just storing mail is simpler than accepting it from outside sources and processing it for delivery. Therefore, it may be possible to get away with a single storage server. However, that is not always the case. MTA MTA MTA MTA

Distributed filesystem on SAN Username-based load balancing Mail storage Single storage server MTA MDA MTA MDA Storage MTA MDA Distributed filesystem on SAN MTA MDA MTA MDA MTA MDA SAN Username-based load balancing The simplest solution is to have all MTAs talk to a single MTA whose job it is to just store all mail. The solution is simple, but won’t scale. Using a distributed file system (e.g. GFS) on the storage system itself, and perhaps placing it on a SAN, the MTAs can deliver directly to shared storage. This solution is complex, but performs very well. Less expensive is to use multiple storage servers, each for a different set of users. This is fairly simple for incoming mail, but complicates mail reading, as different users need to connect to different storage systems. MTA MDA Storage Users a-m MTA MDA Storage MTA MDA Users m-z

Load-balanced MAAs with SAN User-based load balancing Reading mail Load-balanced MAAs with SAN MAA Client L4 Switch MAA MAA SAN MAA Users a-j User-based load balancing The final piece of the puzzle is how to read mail. Users read mail in a variety of ways, but today the most common way to read mail in large systems is through some kind of network protocol such as POP or IMAP. Web-based mail is also common, but is also often IMAP based. In a large system, a single IMAP server is insufficient. Multiple load balanced servers are somehow required. The simplest solution is to place a L4 switch in front of the MAAs. Another possibility, if each MAA handles a different set of users, is to place a L7 switch in front. In this case, the L7 switch will initiate the session, and when the user is known, the session is forwarded to the correct MAA. And multiple servers mean multiple access to mail storage. This can be through a single storage server or though distributed file systems on a SAN. Client L7 Switch MAA Users k-r MAA Users s-z

Centralized example MTA MTA/MSA MUA MUA MTA IMAP MAA L4 Swich MUA SMTP (AUTH) MTA MTA/MSA MUA MUA MTA IMAP MAA L4 Swich MUA IMAP MAA IMAP L4 Swich IMAP MAA Webmail POP MAA This is a complete example of a large, centralized e-mail system. The public-facing MTA is a pair, load balanced by presenting both as MXes for the mail domain. IMAP servers are load balanced using a HA pair of L4 switches. Outgoing mail is sent through a single smarthost. Storage is managed by a HA pair of storage servers, backed by a SAN. Webmail access is through IMAP. Storage Storage SAN

Corporate example (I) MTA MTA L4 Swich MTA L4 Swich MTA MTA Split DNS mx.eng.example.com smtp.example.com MTA MTA L4 Swich MTA L4 Swich mail.sales.example.com MTA MTA eng.example.com MX smtp.example.com. sales.example.com MX smtp.example.com. rnd.example.com MX smtp.example.com. Split DNS eng.example.com MX mx.eng.example.com. sales.example.com MX mail.sales.example.com. rnd.example.com MX pluto.rnd.example.com. This is an example of using split DNS to implement a distributed MTA scheme internally in a company. Hosts outside the company, when they want to send mail to it, will get a single MX for all mail domains, but the corporate mail servers see individual MXes for each internal domain. So when mail is sent, it first hits the single MX, which in turn is set up to relay mail for the internal domains. It, however, sees the individual MXes, and sends mail on to the internal MTAs. For example, mail for john@sales.example.com will first hit mx.example.com, because the sender sees mx.example.com as the MX record for sales.example.com. When mx.example.com receives the message, it will look up the MX record for eng.example.com and see mail.sales.example.com. The mail is then forwarded to sales.example.com. In this example, outgoing mail, storage and whatever can be different for each internal domain.

Corporate example (II) mx.eng.example.com smtp.example.com MTA MTA L4 Swich MTA Address rewrite L4 Swich mail.sales.example.com MTA MTA example.com MX smtp.example.com. Split DNS eng.example.com MX mx.eng.example.com. sales.example.com MX mail.sales.example.com. rnd.example.com MX pluto.rnd.example.com. This is a different example. In this example, all mail to the corporation is adressed to example.com, but internal addresses use subdomains. A lot of companies do things like this. For example, my e-mail address at Sun, when I was there, was officially david.byers@sun.com, but internally I was dbyers@se.sun.com. Suns e-mail gateway rewrote the address both coming and going. Here I’ve shown address rewriting in a separate MTA. There isn’t any real need to do this, but it is possible, and may even be appropriate in some cases. The technique is definitely relevant. You first have an MTA that decides whether the mail should be accepted or not. If it is to be accepted, it’s sent to an internal MTA for further processing. If the mail system tends to reject a lot of mail out of hand, this may allow you to reduce the number of MTAs that do full mail processing, since rejection is offloaded onto a different system. In this example, mail might be addressed to john.doe@example.com. The MX for example.com is smtp.example.com. This MTA decides to accept the mail and forwards it to the rewriting MTA. The address is rewritten to john@sales.example.com. The rewriting MTA looks up the MX for sales.example.com and finds mail.sales.example.com, to which the message is forwarded.

Summary E-mail components MTA, MDA, MUA etc Postfix architecture Address rewriting Content filtering Spam protection Relaying, greylisting Mail system architectures