1. PRINCIPLES – DNS – ARCHITECTURES – SPAM
Electronic Mail
PRINCIPLES – DNS – ARCHITECTURES – SPAM
Dagens föreläsning handlar om katalogtjänster.

2. Mail system components
SMTP
MDA
(Local)
MTA
MDA
(SMTP)
MDA
(SMTP)
MTA
MDA
(Local)
MSA
MSA
SMTP
SMTP
IMAP/POP3
IMAP/POP3
MAA
MUA
MUA
MAA
It’s a mess. Really.
M stands for Mail, A for agent. MDA MTA MSA MAA MUA MRA are standard components. Not all components may be separate pieces of software, and not all need be present at all.
MRA
MUA

3. MUA Mail User Agent Reads and writes e-mail
Writes to MTA using SMTP (usually)
Reads delivered by MDA or retrieved by MRA
Examples
Mozilla Thunderbird
Outlook Express

4. MRA Mail Retrieval Agent Retrieves e-mail from MAA
Makes mail available to MUA
Examples
Fetchmail
Integrated in Thunderbird etc.
MRA is not always used, but sometimes. It’s sort of ”pull” for mail.

5. MAA Mail Access Agent Authenticates MUA/user Reads e-mail from mailbox
Makes available to MUA
Examples
Courier IMAPD

6. MSA Mail Submission Agent Accepts mail from MUA
Prepares and delivers to MTA
Examples
Postfix postdrop+pickup
sendmail-msa

7. MTA Mail Transfer Agent Routes incoming mail
Determines which MDA to send mail to
May alter mail being routed
Examples
Postfix cleanup + qmgr + + trivial-rewrite
Sendmail

8. MDA Mail Delivery Agent Accepts mail from MTA
Delivers mail to local mailbox
Delivers mail to other MTA
Examples
Postfix local, smtp, pipe
mail.local (local delivery)
procmail (local delivery)

9. Mail system components
SMTP
MDA
(Local)
MTA
MDA
(SMTP)
MDA
(SMTP)
MTA
MDA
(Local)
MSA
MSA
SMTP
SMTP
IMAP/POP3
IMAP/POP3
MAA
MUA
MUA
MAA
It’s a mess. Really.
M stands for Mail, A for agent. MDA MTA MSA MAA MUA MRA are standard components. Not all components may be separate pieces of software, and not all need be present at all.
MRA
MUA

10. Path of a message 1. New message written Mail is written in MUA
Mail is submitted to MSA
2. Reception of message at MSA
Authenticate the user
Is user authorized to send mail?
Rewrite headers if needed
Submit message to MTA
Report success to MUA
3. Reception of message at MTA
Is sender valid and permitted
Is recipient valid and permitted
Are contents valid and permitted
Run applicable content filters
Rewrite header
Select appropriate MDA
Submit mail to MDA
If unsuccessful, queue mail for later delivery

11. Path of a message 4. Reception of message at MDA
Send message to remote SMTP server (example)
5. Rcpt of message at remote MTA
Same as step 3
Submit message to MDA
6. Reception of message at MDA
Check applicable filters/rules
Deliver message to local mail store
7. MAA detects new message
Signal new message to MUA
8. Message read in MUA
Retrieve message from MAA
Present message to user

12. Mail system components
SMTP
MDA
(Local)
MTA
MDA
(SMTP)
MDA
(SMTP)
MTA
MDA
(Local)
MSA
MSA
3. Reception of message at MTA
Is sender valid and permitted
Is recipient valid and permitted
Are contents valid and permitted
Run applicable content filters
Rewrite header
Select appropriate MDA
Submit mail to MDA
If unsuccessful, queue mail for later delivery
5. Rcpt of message at remote MTA
Is sender valid and permitted
Is recipient valid and permitted
Are contents valid and permitted
Run applicable content filters
Rewrite header
Select appropriate MDA
Submit mail to MDA
SMTP
SMTP
IMAP/POP3
IMAP/POP3
MAA
MUA
MUA
MAA
It’s a mess. Really.
M stands for Mail, A for agent. MDA MTA MSA MAA MUA MRA are standard components. Not all components may be separate pieces of software, and not all need be present at all.
7. MAA detects new message
Signal new message to MUA
8. Message read in MUA
Retrieve message from MAA
Present message to user
6. Reception of message at MDA
Check applicable filters/rules
Deliver message to local mail store
4. Reception of message at MDA
Send message to remote SMTP server (example)
2. Reception of message at MSA
Authenticate the user
Is user authorized to send mail?
Rewrite headers if needed
Submit message to MTA
Report success to MUA
1. New message written
Mail is written in MUA
Mail is submitted to MSA
MRA
MUA

13. Postfix architecture trivial-rewrite trivial-rewrite local
Incoming queue
Active queue
local
 File, Command
Network 
qmqpd
lmtp
 Network
cleanup
qmgr
Network 
smtpd
smtp
 Network
pickup
virtual
 File
pipe
 Command
Postdrop queue
postdrop
Deferred queue
Local 
sendmail

14. SMTP Simple Mail Transfer Protocol Default: TCP port 25 RFC 2821
ESMTP – Extensions
Text-based protocol
No encryption
No authentication
Example
%telnet mail.ida.liu.se 25
Trying
Connected to mail.ida.liu.se.
Escape character is '^]'.
220 mail.ida.liu.se ESMTP Sendmail /8.13.3; Mon, 29 Aug :34: (CEST)
EHLO lap-65.ida.liu.se
250-mail.ida.liu.se Hello gedrix.ida.liu.se [ ], pleased to meet you
MAIL
Sender ok
RCPT
Recipient ok
DATA
354 Enter mail, end with "." on a line by itself
From: David Byers
To: Samizdat Info
Date: Mon, 29 Aug :38:
Subject: Testing SMTP
Body text. .
j7TBYgXf Message accepted for delivery
QUIT
portofix.ida.liu.se closing connection

15. The role of DNS Problem: which server to contact for a mail address?
Solution: look up MX record in DNS
Example:
example.com MX 10 mail1.example.com.
example.com MX 10 mail2.example.com.
example.com MX 20 mx.nodomain.edu.
Explain what MX records are good for. Talk about SRV records as a generalization of the same. Talk about what happens if there is no MX record. Note that MX records are valid for the name only, not subdomains. Explain the priority field.

16. Structure of a message Envelope From SMTP dialog Header
Message metadata
Specified in RFC 2822
Body
Sequence of ASCII characters
Separated from body by CRLF
Syntax
message::= headers CRLF body
headers ::= header headers | e
header ::= name ”:” value CRLF
Return-Path:
Delivery-Date: Fri May 14 08:38:
Received: from diag7.ida.liu.se (diag7.ida.liu.se [ ])
by portofix.ida.liu.se ( / ) with ESMTP id i423760;
Fri, 14 May :38: (MEST)
Received: (from
by diag7.ida.liu.se ( Sun/ /Submit) id i401197;
Fri, 14 May :38: (CEST)
Date: Fri, 14 May :38:
From: Andreas Johansson
To: David Byers
Cc:
Subject: Re: =?ISO ?Q?N=E4tverksproblem?=
Message-Id:
In-Reply-To:
References:
Organization: =?ISO ?Q?Link=F6pings?= universitet
X-Mailer: Sylpheed version (GTK ; sparc-sun-solaris2.9)
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO
X-Virus-Scanned: clamd / ClamAV 0.70, clamav-milter 0.70j
X-Spam-Flag: NO
X-Scanned-By: milter-spamc/ ( [ ]); pass
Walk through the important headers:
Received
Note header rewriting has taken place
Note that from line means nothing
Note X headers

17. Some things the MTA does
Header rewriting
Addition of received header
Addition of Message-ID
Address rewriting
Example rewriting
Removal of host name
Addition of domain name
davby  david.byers
Mail routing
Find server for recipient address
Queue mail for submission
Re-submit failed messages
Send delivery status notifications
SMTP example
Find MX records for address
Try records in ascending order
Try A record for address

18. Postfix address rewriting
Rewrite addresses to standard form
Canonical address mapping
Address masquerading
Automatic BCC recipients
Virtual aliasing
Resolve address to destination
Mail transport switch
Relocated users table
Generic mapping table
Local alias database
Local per-user .forward files
Local catch-all address
smtpd
qmqpd
pickup
cleanup
trivial-rewrite
(rewrite)
Incoming queue
Deferred queue
Active queue
qmgr
trivial-rewrite
(resolve)
smtpd
lmtp
local

19. Postfix address rewriting
Rewrite addresses to standard form
Canonical address mapping
Address masquerading
Automatic BCC recipients
Virtual aliasing
Resolve address to destination
Mail transport switch
Relocated users table
Generic mapping table
Local alias database
Local per-user .forward files
Local catch-all address
smtpd
qmqpd
pickup
Examples
user%domain ->
user ->
->
cleanup
trivial-rewrite
(rewrite)
Incoming queue
Deferred queue
Active queue
qmgr
trivial-rewrite
(resolve)
smtpd
lmtp
local

20. Postfix address rewriting
Rewrite addresses to standard form
Canonical address mapping
Address masquerading
Automatic BCC recipients
Virtual aliasing
Resolve address to destination
Mail transport switch
Relocated users table
Generic mapping table
Local alias database
Local per-user .forward files
Local catch-all address
smtpd
qmqpd
pickup
cleanup
trivial-rewrite
(rewrite)
Examples
user -> first.last
->
->
first.last -> user
Incoming queue
Deferred queue
Active queue
qmgr
trivial-rewrite
(resolve)
smtpd
lmtp
local

21. Postfix address rewriting
Rewrite addresses to standard form
Canonical address mapping
Address masquerading
Automatic BCC recipients
Virtual aliasing
Resolve address to destination
Mail transport switch
Relocated users table
Generic mapping table
Local alias database
Local per-user .forward files
Local catch-all address
smtpd
qmqpd
pickup
cleanup
trivial-rewrite
(rewrite)
Incoming queue
Deferred queue
Active queue
Examples
->
qmgr
trivial-rewrite
(resolve)
smtpd
lmtp
local

22. Postfix address rewriting
Rewrite addresses to standard form
Canonical address mapping
Address masquerading
Automatic BCC recipients
Virtual aliasing
Resolve address to destination
Mail transport switch
Relocated users table
Generic mapping table
Local alias database
Local per-user .forward files
Local catch-all address
smtpd
qmqpd
pickup
cleanup
trivial-rewrite
(rewrite)
Incoming queue
Deferred queue
Active queue
qmgr
trivial-rewrite
(resolve)
Examples
->
smtpd
lmtp
local

23. Postfix address rewriting
Rewrite addresses to standard form
Canonical address mapping
Address masquerading
Automatic BCC recipients
Virtual aliasing
Resolve address to destination
Mail transport switch
Relocated users table
Generic mapping table
Local alias database
Local per-user .forward files
Local catch-all address
smtpd
qmqpd
pickup
cleanup
trivial-rewrite
(rewrite)
Incoming queue
Deferred queue
Active queue
qmgr
trivial-rewrite
(resolve)
smtpd
lmtp
local
Examples
root -> user
first.last -> user

24. More things the MTA does
Content filtering
Scan for virus content
Check validity of attachments
Check size of messages
Check for spam
Common spam checks
Keyword/feature filters
IP address blacklists
Greylisting
Authorization
Check submitting host IP
Check from address
Check recipient address
Check contents
Typically permitted
Mail from local systems
Mail to local recipients
Mail from trusted hosts to any recipient

25. Postfix content filters
Three flavors
Built-in content inspection
After queue content filter
Before queue content filter
”Body checks”, ”Header checks”
Regexp checks on message
For stopping specific problems
pickup
qmqpd
smtpd
(filter)
cleanup
bounce
local (forwarded)
Incoming queue
Postfix has three types of content filters. Built in content inspection is known as header and body checks, and are regular expressions that are matched against the message. These are to stop specific problems, not for general spam filtering. Since this filtering is done on-line, if it takes too much time, the cleanup process will take up all CPU time, and the number of simultaneous SMTP connections will reach the maximum.

26. Postfix content filters
Three flavors
Built-in content inspection
After queue content filter
Before queue content filter
After mail has been accepted
Heavy-duty filtering
Need to avoid spurious DSNs
pickup
qmqpd
smtpd
(filter)
cleanup
bounce
local (forwarded)
Postfix postdrop
Postfix queues
Postfix has three types of content filters. After queue content filters are run after messages are accepted for delivery. This allows the SMTP dialog to finish before the content filter runs. In systems that accept large amounts of mail, this may be necessary to avoid running out of available sockets. However, after queue content filters have no reliable way of notifying the senter of problems.
Postfix sendmail
qmgr
Content filter
pipe
smtp
local

27. Postfix content filters
Three flavors
Built-in content inspection
After queue content filter
Before queue content filter
Before mail is accepted
Heavy-duty filtering
Only for low-volume sites
smtpd
Content filter
smtpd
(filter)
cleanup
Postfix has three types of content filters. Before-queue content filters run while the SMTP dialog is active, so the content filter can reject the message as part of the SMTP dialog. The obvious problem is that the content filter increases SMTP response time. Under high load, the delay may be sufficient to cause the SMTP client to time out.
Incoming queue

28. Spam protection
Relay restrictions
Greylisting
DNS-based blocklists

29. SMTP Restrictions Denying relay Greylisting in action
220 solzhenitsyn.samizdat.se ESMTP Postfix
HELO gedrix.ida.liu.se
250 solzhenitsyn.samizdat.se
MAIL
250 Ok
RCPT
554 Relay access denied
Greylisting in action
220 portofix.ida.liu.se ESMTP Sendmail /8.13.3
HELO metzengerstein.visit.se
250 portofix.ida.liu.se Hello metzengerstein.visit.se [ ]
MAIL
Sender ok
RCPT
Greylisting in action, please come back in 00:10:00
SMTP restrictions are restrictions implemented by the SMTP server. Typically they look at the headers, the network connection and the SMTP dialog only, and are very lightweight. For example, a common restriction is to refuse mail from external IP addresses that is not addressed to local addresses.

30. DNS-based blocklists Blocklists work like in-addr.arpa lookup
%host -t TXT l1.spews.dnsbl.sorbs.net
l1.spews.dnsbl.sorbs.net TXT
"! [1] XPD Limited, see
%host -t TXT dnsbl.sorbs.net
dnsbl.sorbs.net TXT "Spam Received See:
Talk about how they work. Talk about the different policies of blocklists and that this means you have to choose carefully which ones to actually use.

31. Architectures

32. Client-side outgoing mail
Client has an MTA
Client MTA queues mail
Client has no MTA
If central MTA is down, no mail can be sent
Client
MTA
MUA
Client
MUA
The part of the system closes to the user is the MUA. Here there are a number of options.
Traditionally, unix systems have an MTA on each client. This allows the client to queue outgoing mail regardless of whether the destination is ready to accept it or not. Having an MTA on the client increases complexity somewhat, but also increases robustness. Often, the integrated MTA will not be an SMTP server.
Today, most users do not have a local MTA. Instead, they use MUAs that connect directly to an external MTA. This way, if the central MTA is unavailable, no mail can be sent. On the other hand, users will get immediate feedback on any problems in the system.
MTA
Client
MUA

33. Outgoing mail Smarthost All mail through one MTA Central configuration
Failover
Better reliability
Load balancing
Better performance
Client
Client
MTA
Client
Client
MTA
Client
MTA
Client
Most architectures are a variant of a smarthost architecture. Here, all main from within a network is transferred to a dedicated mail server, called the smarthost, which is responsible for things like address rewriting, filtering and so forth. The MTAs on each individual client may be as simple as a program that can communicate with the smarthost using SMTP – there is no need for an SMTP server on each client (although there might be one).
This architecture permits centralized administration and very simple client configuration. However, the basic smarthost system has a SPOF in the smarthost. To improve reliability, failover needs to be employed. There are several ways to accomplish this, but the most reliable method is to set up two servers that monitor each other. When one goes down, the other takes over its IP address and possibly MAC address.
Failover doesn’t address scalability very well. A layer 7 switch, or better still a pair of them, can balance the load from all clients over all MTAs. The layer 7 switches can also monitor the MTAs to avoid those that are offline.
Client
Client
Client
Client
L4 Switch
MTA
Client
MTA
Client
L4 Switch
MTA
Client
Client
Client

34. One front-end multiple backends
Incoming mail
Multiple MTAs
MTA
MDA
MTA
MDA
Load-balanced MTAs
MTA
MDA
MTA
MDA
L4 Switch
MTA
MDA
L4 Switch
MTA
MDA
One front-end multiple backends
For incoming mail, there are several concerns: the load must be managed; the service must be reliable; configuration must be sensible; and so on.
Organizations with lots of mail will want to have multiple MTAs. This can easily be accomplished by publishing multiple MX records with equal preference in DNS. It can also be accomplished using an L7 switch. An L7 switch will guaranteee even load balancing, which DNS tricks won’t. On the other hand, an L7 switch is an additional complication.
In some systems, a hierarchy of MTAs will be used. The top-level MTA transfers mail to MTAs deeper in the hierarchy. This is generally useful for an organization that wants one public-facing MTA, but in reality has lots of internal systems. How mail is distributed between internal MTAs depends on the situation. One possibility is to have multiple subdomains. Another is to have the public-facing MTA rewrite addresses into multipls subdomains. There are other ways too.
In these architectures, we have multiple MTAs, but all mail generally needs to go in a single mail storage system. Luckily, just storing mail is simpler than accepting it from outside sources and processing it for delivery. Therefore, it may be possible to get away with a single storage server. However, that is not always the case.
MTA
MTA
MTA
MTA

35. Distributed filesystem on SAN Username-based load balancing
Mail storage
Single storage server
MTA
MDA
MTA
MDA
Storage
MTA
MDA
Distributed filesystem on SAN
MTA
MDA
MTA
MDA
MTA
MDA
SAN
Username-based load balancing
The simplest solution is to have all MTAs talk to a single MTA whose job it is to just store all mail. The solution is simple, but won’t scale.
Using a distributed file system (e.g. GFS) on the storage system itself, and perhaps placing it on a SAN, the MTAs can deliver directly to shared storage. This solution is complex, but performs very well.
Less expensive is to use multiple storage servers, each for a different set of users. This is fairly simple for incoming mail, but complicates mail reading, as different users need to connect to different storage systems.
MTA
MDA
Storage
Users a-m
MTA
MDA
Storage
MTA
MDA
Users m-z

36. Load-balanced MAAs with SAN User-based load balancing
Reading mail
Load-balanced MAAs with SAN
MAA
Client
L4 Switch
MAA
MAA
SAN
MAA
Users a-j
User-based load balancing
The final piece of the puzzle is how to read mail. Users read mail in a variety of ways, but today the most common way to read mail in large systems is through some kind of network protocol such as POP or IMAP. Web-based mail is also common, but is also often IMAP based.
In a large system, a single IMAP server is insufficient. Multiple load balanced servers are somehow required. The simplest solution is to place a L4 switch in front of the MAAs. Another possibility, if each MAA handles a different set of users, is to place a L7 switch in front. In this case, the L7 switch will initiate the session, and when the user is known, the session is forwarded to the correct MAA.
And multiple servers mean multiple access to mail storage. This can be through a single storage server or though distributed file systems on a SAN.
Client
L7 Switch
MAA
Users k-r
MAA
Users s-z

37. Centralized example MTA MTA/MSA MUA MUA MTA IMAP MAA L4 Swich MUA
SMTP (AUTH)
MTA
MTA/MSA
MUA
MUA
MTA
IMAP MAA
L4 Swich
MUA
IMAP MAA
IMAP
L4 Swich
IMAP MAA
Webmail
POP MAA
This is a complete example of a large, centralized system. The public-facing MTA is a pair, load balanced by presenting both as MXes for the mail domain. IMAP servers are load balanced using a HA pair of L4 switches. Outgoing mail is sent through a single smarthost. Storage is managed by a HA pair of storage servers, backed by a SAN. Webmail access is through IMAP.
Storage
Storage
SAN

38. Corporate example (I) MTA MTA L4 Swich MTA L4 Swich MTA MTA Split DNS
mx.eng.example.com
smtp.example.com
MTA
MTA
L4 Swich
MTA
L4 Swich
mail.sales.example.com
MTA
MTA
eng.example.com MX smtp.example.com.
sales.example.com MX smtp.example.com.
rnd.example.com MX smtp.example.com.
Split DNS
eng.example.com MX mx.eng.example.com.
sales.example.com MX mail.sales.example.com.
rnd.example.com MX pluto.rnd.example.com.
This is an example of using split DNS to implement a distributed MTA scheme internally in a company. Hosts outside the company, when they want to send mail to it, will get a single MX for all mail domains, but the corporate mail servers see individual MXes for each internal domain. So when mail is sent, it first hits the single MX, which in turn is set up to relay mail for the internal domains. It, however, sees the individual MXes, and sends mail on to the internal MTAs.
For example, mail for will first hit mx.example.com, because the sender sees mx.example.com as the MX record for sales.example.com. When mx.example.com receives the message, it will look up the MX record for eng.example.com and see mail.sales.example.com. The mail is then forwarded to sales.example.com.
In this example, outgoing mail, storage and whatever can be different for each internal domain.

39. Corporate example (II)
mx.eng.example.com
smtp.example.com
MTA
MTA
L4 Swich
MTA
Address rewrite
L4 Swich
mail.sales.example.com
MTA
MTA
example.com MX smtp.example.com.
Split DNS
eng.example.com MX mx.eng.example.com.
sales.example.com MX mail.sales.example.com.
rnd.example.com MX pluto.rnd.example.com.
This is a different example. In this example, all mail to the corporation is adressed to example.com, but internal addresses use subdomains. A lot of companies do things like this. For example, my address at Sun, when I was there, was officially but internally I was Suns gateway rewrote the address both coming and going.
Here I’ve shown address rewriting in a separate MTA. There isn’t any real need to do this, but it is possible, and may even be appropriate in some cases. The technique is definitely relevant. You first have an MTA that decides whether the mail should be accepted or not. If it is to be accepted, it’s sent to an internal MTA for further processing. If the mail system tends to reject a lot of mail out of hand, this may allow you to reduce the number of MTAs that do full mail processing, since rejection is offloaded onto a different system.
In this example, mail might be addressed to The MX for example.com is smtp.example.com. This MTA decides to accept the mail and forwards it to the rewriting MTA. The address is rewritten to The rewriting MTA looks up the MX for sales.example.com and finds mail.sales.example.com, to which the message is forwarded.

40. Summary E-mail components MTA, MDA, MUA etc Postfix architecture
Address rewriting
Content filtering
Spam protection
Relaying, greylisting
Mail system architectures