1. Design and Management of Email Service
張傑生Jason Chang
Computer Center
National Taiwan University
2005/07/08
北區區網訓練課程

2. Outline A brief view of NTU email service
Introduction to the architecture and operation of SMTP
Design of a suitable system
Webmail solutions
Postfix and simple configuration samples
Spam and virus filtering
Conclusion

3. Overview
Electronic mail service has already evolved into one of the major Internet applications.
It is not only fundamental, but also a must.
Users may become impatient when mails were delayed, not to mention failed to access their s.
Imagine we meet the situation of power failure or cut of water supply

4. A Brief View of NTU Email Service
Current Status
statistics
Average daily incoming: 800,000
Truly delivered: 200,000(25%)
Account statistics
Faculty/staff/official use: 4200
Student: 30000
Alumni: 24500
Account policy
Lifelong preserved for retired faculties/staffs and alumni
Server design
Incoming relay/filter: 3
Outgoing relay: 1
Webmail: 2
Pop3: 5 for students and 1 for others

5. A Brief View of NTU Email Service
Problems we faced
Account creation and remove
Non-decreasing number of users
Spam
What we have done
Database synchronization with campus mis systems
Single Sign On
Totally encrypted transmission
Spam and virus filtering
Ugly but work architecture for scalability concern

6. Architecture of a Simple Mail System
Consists of the following components
MTA - Mail transfer agent
Sending and forwarding s
Server end
MDA - Mail delivery agent
Delivering s to recipients’ mailbox
Pop3/Imap4 Daemons
For users to download their mailboxs
MUA - Mail user agent
Reading and composing s
Client end

7. Architecture of a Simple Mail System

8. Protocols Used for Mail System
For computer programs to communicate with each other
Similar to languages that human beings speak
SMTP
Simple Mail Transfer Protocol
Too simple to provide any “advanced features”
Authentication
Authorization
POP3
Post Office Protocol version 3
Simple
IMAP4
Internet Message Access Protocol version 4
Fully compatible with internet message standards, e.g. MIME.
Allow messages to be accessed from more than one computer.
Provide support for online, offline, and disconnected modes.
Multiple and share folders.

9. Mail Forwarding Between Servers

10. How to Find the Way to the Destination?
How do we find the way to

11. DNS: The Key to All Internet Services
Query DNS server by the address part of
Query for MX records
Query for A records
Give up!
Not required to exist a real machine hosting the address.(ntu.edu.tw)
Adding MX records to improve query performance and to reduce unnecessary query overhead.

12. DNS: Query Results Precedence value of MX records
Lower values with higher priority

13. DNS: Query Results
When MX record is absent, second query is required to issue.

14. The Use of Mail Relay Gateway
Load sharing and balance
Prevent the occurrence of single point of failure
Hide your backend mail servers and prevent from outside attack
Frequently used when you have M$ Exchange servers
Make use of multiple network link connections (different ISPs)
Spam and virus filtering
The easiest way for administrators to release current bottlenecks and to leverage performance.
It’s the way how most commercial products work.

15. The Architecture of Mail Relay Gateways

16. Planning of a Modern Mail System
This is what system administrators care.
Scalability
User / spool capacity
Virtual user management
Availability
Multiple relay servers
Multiple pop3 servers
Security
Avoid creating system accounts
No access to shell
Performance
Make the right decision
Architecture
OS/Software

17. Features Required By Modern Mail Systems
This is what end users care.
Ease of use interface
IMAP4
Webmail
Security
Authentication
Encrypted transmission
Spam and virus filtering
The only effective way to prevent from virus infection is to remind and force your users to install antivirus softwares.
Stop using M$ systems is alternative solution.

18. Scalability of a Mail System
Traditional way:
Additional servers with different address.
ms1.cc.ntu.edu.tw
ms2.cc.ntu.edu.tw
ms3.cc.ntu.edu.tw
Ugly and not professional
Modern and Clever way:
Multiple backend servers
Make use of relay gateways to dispatch and deliver mails
A uniform address for all of the users.
Virtual user management by central database

19. A Scalable Mail System Architecture

20. Improving Scalability and Availability By Adding Multiple POP3 Servers
POP3/IMAP4 accesses are more frequent than incoming mail delivery.
Users may easily aware of the failure of pop3 servers in comparison to smtp/relay servers. (in:out = 20:1)
NFS protocol is used to share the common storage.
Mail delivery efforts can be dealt by other servers.
Totally rely on the stability of storage server.
Single point of failure
Could be replaced by high-end NFS servers or NetApp filers.

21. Architecture of Multiple POP3 Servers

22. NFS Concerns And Bottlenecks
NFS has its own limitation.
Do not expect NFS to scale well.
Use gigabit network with high quality switch.
UDP is preferred over TCP when network environment is local and qualified.
Consider to apply the Maildir format
Avoid NFS lock problem
Severe enough in Linux and BSD systems
Spin lock and dead lock may block further accesses
Manual interference often requires

23. Make the Right Decision For the Promise of Performance
Capacity planning
Reasonable system design and architecture
Operating system
Linux, BSD
Always the best and wise choice.
Solaris
It’s been called Slow……laris. M$
Are you kidding?

24. Make the Right Decision For the Promise of Performance
MTA
Postfix
Our suggestion!
Sendmail
Not scalable!
Lack of new features.
Not efficiency in resource management.
Rich of documents and tutorials
Qmail
Termination of development.
Exim
Immature.
M$ Exchange
Seeking trouble for yourself?
POP3/IMAP4
UW-IMAP
Courier-IMAP
Supports Maildir format.
Dovecot
Supports both mbox and Maildir format.
Still under development.
Cyrus
Proprietary spool format.
Not easy to convert between other formats.

25. Webmail: The Trend of Browser Based Mail Reader
Your determination depends on the system architecture.
IMAP4 based
Access the mailbox via IMAP4 protocol locally or remotely.
Flexible, scalable
Speed may be limited by protocol overhead and network transmission.
Horde IMP project
Locally direct access
Access the mailbox locally, proprietary protocol.
Could only run on local mail storage server.
NFS could be applied when deploying multiple servers
Extremely fast!! iff system not overloaded.
Not scale well when you want to expand.
Openwebmail
Excellent project developed by NCKU, Taiwan
Marvelous support to Chinese encodings
Only support mbox format currently

26. Webmail Architecture – IMAP4 based

27. Webmail Architecture – Locally Direct Access

28. Postfix: The Modern and Advance MTA
Developed by Wietse Venema
Features
Easy replacement of existing sendmail system by binaries with same file name and functionality . e.g. sendmail, newaliases
Human readable configuration file
Multiple small programs with limited execution privilege
Multiple queues to avoid block of normal mails
Backend database lookup table supported
Better support for virtual host
Virtual user system supported
Content filtering interface
Simple spam check/block mechanism

29. Postfix - the Big Picture
Yellow ellipsoids are mail programs.
Yellow boxes are mail queues or files.
Blue boxes are lookup tables.

30. Postfix Installation Debian Linux Redhat/Fedora Linux BSD
apt-get install postfix-tls libsasl7 libsasl-modules-plain courier-imap
Redhat/Fedora Linux
rpm -ivh postfix-2.2.x.i386.rpm
rpm -ivh cyrus-sasl i386.rpm
BSD
Use the ports system
/usr/ports/mail/postfix
/usr/ports/security/cyrus-sasl2
/usr/ports/mail/courier-imap

31. Post Installation Postfix Configuration
master.cf
Similar to inetd.conf
Control the behavior of small programs
In contrast against sendmail, with one binary and one config file
main.cf
The main configuration of the mail system
In general cases, no modification is required for a simple setup.
Commands to control the postfix program
postfix start
postfix stop
postfix reload

32. Test For Receiving Email
Send an to yourself.
Check system logs
/var/log/maillog
An simple test from to

33. Frequently Used Configuration Options In main.cf
myhostname = mail.cc.ntu.edu.tw
mydomain = cc.ntu.edu.tw
myorigin = $mydomain
mydestination = $myhostname, localhost.$mydomain
On a mail domain gateway, you should also include $mydomain.
mynetworks_style = host
relay_domains = $mydestination
home_mailbox = Maildir/
message_size_limit =
smtpd_recipient_limit = 1000

34. Configure to Enable SSL
main.cf
smtpd_enforce_tls = yes
smtpd_use_tls = yes
smtpd_tls_cert_file = /usr/local/etc/ssl/smtp.cert
smtpd_tls_key_file = /usr/local/etc/ssl/smtp.key
smtpd_tls_CAfile = /usr/local/etc/ssl/ntuca.crt
smtpd_tls_loglevel = 1
smtpd_tls_received_header = yes
smtp_enforce_tls = yes
smtp_tls_cert_file = $smtpd_tls_cert_file
smtp_tls_key_file = $smtpd_tls_key_file
smtp_tls_CAfile = $smtpd_tls_CAfile
smtp_tls_loglevel = 1
smtp_use_tls = yes
smtp_tls_note_starttls_offer = yes
tls_random_exchange_name = /var/run/prng_exch
tls_random_source = dev:/dev/urandom
tls_daemon_random_source = dev:/dev/urandom
master.cf
tlsmgr fifo - - n tlsmgr

35. Test for SSL/TLS Connection
telnet to mail server and check for welcome messages.

36. Test for SSL/TLS Connection
Send an to yourself.
From mx1 to ccms
Check system logs
/var/log/maillog

37. SMTP/Authentication
Traditional mail servers could only determine their relay policy based on connected ip addresses.
Insufficient in face of the modern network
DHCP, NAT
ADSL and dialup users
Roaming users
Mobile, wireless
SMTP Authentication
Username/password authentication for each mail delivery.
Reduce the chance to relay spam mails for anonymous.

38. Setting Up SMTP/Authentication
Install Cyrus-SASL2 library
It could be configured to authenticate via /etc/passwd, PAM, LDAP, backend databases
Modify /usr/local/lib/sasl2/smtpd.conf
pwcheck_method: saslauthd
mech_list: plain login
Make sure saslauthd is running.

39. Configure to Enable SMTP/Authentication
Since the username/password negotiation is not encrypted. We must configure our server to accept incoming mail authentication only on TLS encrypted connections.
main.cf
smtpd_sasl_auth_enable = yes
smtpd_sasl_security_options = noanonymous
smtpd_tls_auth_only = yes
smtpd_recipient_restrictions = reject_unknown_recipient_domain, reject_non_fqdn_recipient, permit_sasl_authenticated, reject_unauth_destination
master.cf
smtps inet n - n - - smtpd -o smtpd_tls_wrappermode=yes
The smtpd will listen on port 465 instead of 25.

40. Test For SMTP/Authentication Connection
The username/password is required to be encoded in BASE64.
The Perl MIME::Base64 module could be installed.
/usr/ports/converters/p5-MIME-Base64
perl -MMIME::Base64 -e 'print encode_base64("user\0user\0passwd");‘
dGVzdDEAdGVzdDEAcGFzczE=

41. Test For SMTP/Authentication Connection
Send an to yourself.
Check system logs
/var/log/maillog

42. SMTP Transaction Example

43. Email Headers Envelope header Content Text
Negotiated between mail servers
Typical tricks that SPAM mail plays
Why do I receive mails that are not destined to me?
Content Text
Mail servers will not examine the content in general cases.

44. Real World Snail Mail Example
Postman could only deliver the mail by envelope information.
Sender address is not verified when delivery.
Trick of SPAM mails.
Content may be irrelevant or conflict with envelope.

45. Postfix: Spam Filtering Functions
SMTP access restriction
Refer to the previous SMTP transaction example
Restriction list name
Effect of REJECT or DEFER result
smtpd_client_restrictions
Reject all client commands
smtpd_helo_restrictions
Reject HELO/EHLO information
smtpd_sender_restrictions
Reject MAIL FROM information
smtpd_recipient_restrictions
Reject RCPT TO information

46. Postfix: Spam Filtering Functions
Access maps
Similar to sendmail access map
Control relay permission based on IP addresses
main.cf
smtpd_client_restrictions = check_client_access hash:/usr/local/etc/postfix/access
Execute postmap access after update
Dynamic IP ranges should be blocked

47. Postfix: Spam Filtering Functions
Realtime black list
Use with caution
The definition and meaning of each rbl
Too many TW hosts are on the lists
main.cf
maps_rbl_domains = bl.spamcop.net, relays.ordb.org, orbs.dorkslayers.com
smtpd_client_restrictions = reject_maps_rbl

48. Postfix: Spam Filtering Functions
SMTP access restriction
smtpd_client_restrictions
maps_rbl_domains = bl.spamcop.net, relays.ordb.org, orbs.dorkslayers.com
smtpd_client_restrictions = reject_rbl_client relays.ordb.org reject_rbl_client bl.spamcop.net reject_maps_rbl check_client_access regexp:/usr/local/etc/postfix/access reject_unauth_pipelining reject_unknown_client (use with caution)
smtpd_helo_restrictions
smtpd_helo_required = yes
smtpd_helo_restrictions = reject_invalid_hostname reject_unknown_hostname reject_non_fqdn_hostname

49. Sample Rejected Logs

50. Postfix: Spam Filtering Functions
SMTP access restriction
smtpd_sender_restrictions
smtpd_sender_restrictions = reject_unknown_sender_domain reject_non_fqdn_sender check_sender_access hash:/etc/postfix/sender-access
Useful to block SPAM mails that fake envelope mail from header

51. Postfix: Spam Filtering Functions
SMTP access restriction
smtpd_recipient_restrictions
smtpd_recipient_restrictions = check_recipient_access regexp:/usr/local/etc/postfix/recp-access reject_unknown_recipient_domain reject_unauth_destination permit_sasl_authenticated permit_mx_backup

52. Postfix: Spam Filtering Functions
Header check
Both envelope and content headers
main.cf header_checks = regexp:/usr/local/etc/postfix/header

53. Postfix: Spam Filtering Functions
Simple content check
main.cf
body_checks = regexp:/usr/local/etc/postfix/body
Content filtering interface
Amavisd-new as an example

54. Virus and Spam Mail Virus mail Spam mail
Contain virus infected attachment
How do we determine if the mail is virus infected?
Simple and easy. Base on the inspection and judgment of antivirus software.
Spam mail
UCE/UBE
Unsolicited Commercial
Unsolicited Bulk
How do we determine if the mail is a Spam mail?
Difficult. It’s not a yes/no problem.
One person’s meat may be another’s poison.

55. Issues When Dealing With SPAM
Do not block or discard users’ mail unless you have the authorization and delegation.
The only safe and acceptable assistant you can provide is to score and tag each mail and let your users to do the rest themselves.
To warn or not to warn the senders/recipients is a dilemma!
Virus and spam senders nowadays always fake the from address.
System notification mails may be another kind of SPAM. The result will be blamed by your users and blocked by other mail servers.

56. Greylisting
Recently proposed (2003) method to block spam and virus mails.
RFC requires MTA to be capable to support queue and retry of temporarily undeliverable mails.
Spam and virus always fire and forget.
For efficiency concern, usually they do not retry.
How it works?
In brief, block every delivery at its first try.
For each mail delivery, check if the the triplet(source ip, sender, recipient) was seen recently.
If yes, pass the mail.
If not, reject the delivery with “450 Please retry later.”

57. Greylisting Drawback Risk Suggestion
Mail delivery will suffer a delay ranging from several minutes to hours.
Depending on the remote MTA configuration.
Risk
Sites that do not retry their mail delivery.
Some bank notifications were observed to try only once.
Mail sending programs that process the SMTP transaction themselves without calling the “sendmail” MTA interface.
Suggestion
Whitelisted known hosts to prevent from delay and loss of mail.

58. Greylisting Benefit Effectively reduce the volume of spam and virus.
Effectively reduce both cpu and disk load of servers.

59. Softwares To Block/Filter Spam and Virus
Postfix
Content filter interface
Postgrey
Greylisting policy server
Amavisd-new
Clam Antivirus
A GPL virus scanner
SpamAssassin

60. Software Architecture for Filtering

61. Amavisd-new
A high performance interface between MTA and content checkers.
Calling external antivirus programs to do virus scanning.
Calling external spamassassin program to do spam level determination.
CPU intensive workloads.
Can be flexibly configured to pass, discard, or quarantine mails based on user defined policy.
Pass spam mails with score > 10 with subject prepended the *** SPAM *** keyword.
Quarantine spam mails with score > 20.
Discard spam mails with score > 30.
Quarantine virus mails.

62. Spamassassin Spam level scoring software.
Rich set of tests to identify various spam signatures.
Keywords, bad headers, encodings
Use bayesian analysis to help scoring.
Training the bayesian database using know spam and ham mails.
Default to enable the auto-learn feature.
Calling external programs to check if the mail was a known spam.
Use hash of mail content as the query key.
Razor, DCC, Pyzor.

63. Spamassassin
RBL(realtime black list) look up based on sender ip address.
RBL may contains too many ill-administrated sites.
Use the result as an addition of spam score.
Do not block remote sites depend solely on RBL.
SURBL(Spam URI realtime black list) look up based on the URIs within the content of mail.
Spammers may keep changing their sending IP addresses.
The URIs in the content may be the final destination the advertisement want people to visit.

64. Postfix: Content Filter Configuration
master.cf
smtp-amavis unix - - y/n - 2 smtp o smtp_data_done_timeout= o smtp_send_xforward_command=yes o disable_dns_lookups=yes
:10025 inet n - y/n - - smtpd o content_filter= o local_recipient_maps= o relay_recipient_maps= o smtpd_restriction_classes= o smtpd_client_restrictions= o smtpd_helo_restrictions= o smtpd_sender_restrictions= o smtpd_recipient_restrictions=permit_mynetworks,reject o mynetworks= / o strict_rfc821_envelopes=yes o smtpd_error_sleep_time= o smtpd_soft_error_limit= o smtpd_hard_error_limit=1000
main.cf
content_filter = smtp-amavis: :10024

65. Amavisd-new Configuration
amavisd.conf
$max_servers = 30; # number of pre-forked children
@av_scanners = ( ….
$final_virus_destiny = D_DISCARD; # (defaults to D_BOUNCE)
$final_banned_destiny = D_BOUNCE; # (defaults to D_BOUNCE)
$final_spam_destiny = D_DISCARD; # (defaults to D_REJECT)
$final_bad_header_destiny = D_PASS; # (defaults to D_PASS), D_BOUNCE suggested
$QUARANTINEDIR = '/var/virusmails/infected';
$sa_tag_level_deflt = 1; # add spam info headers if at, or above that level
$sa_tag2_level_deflt = 9; # add 'spam detected' headers at that level
$sa_kill_level_deflt = 20; # triggers spam evasive actions
$sa_dsn_cutoff_level = 20; # spam level beyond which a DSN is not sent
$sa_quarantine_cutoff_level = 30;
Raise the tag2 value to avoid removing users’ mail.

66. SpamAssassin Configuration
Built-in tests
local.cf
ok_languages en ja zh
ok_locales en ja zh
score SUBJ_ILLEGAL_CHARS 0
score FROM_ILLEGAL_CHARS 0
score HEAD_ILLEGAL_CHARS 0
score CHARSET_FARAWAY 1.0
score CHARSET_FARAWAY_HEADER 1.0
score MIME_CHARSET_FARAWAY 1.0
header NTU_SMTP Received =~ /mx3.cc.ntu.edu.tw \(relay5.tp1rc.edu.tw \[ \]\) \(using TLSv1 with cipher DHE-RSA-AES256-SHA \(256\/256 bits\)\) .* by mx1.cc.ntu.edu.tw \(Postfix\)/
score NTU_SMTP -15.0
describe NTU_SMTP mx3.cc.ntu.edu.tw
header HINET_MSR Received =~ /msr\d{1,2}.hinet.net \(msr\d{1,2}.hinet.net \[ \d{1,2}\]\) by mx1.cc.ntu.edu.tw \(Postfix\)/
score HINET_MSR -10.0
describe HINET_MSR msr.hinet.net

67. Sample Log of Postfix
Each incoming mail may have up to two transaction logs.

68. Sample Log of Amavisd

69. Sample Headers of User’s Mail

70. Sample Headers of User’s Mail

71. Sample Headers of User’s Mail

72. Sample Screenshots of Outlook Express

73. Make Use of the Filter Rules on your MUA
Outlook Express
Can only filter based on limited headers.
Subject, Sender, etc.
Configure to move mails having subjects beginning with *** SPAM *** to an SPAM folder.
Outlook, Netscape, Thunderbird
You can filter mails based on any headers.
Determine your own spam level threshold referring the X-Spam-Level: ******

74. Conclusion
Make use of public domain, open source software to construct your own mail system.
Official web sites, mailing lists, and google are always your best friends.
Feedback and contribute as possible as you can.