1. Spam Reduction Techniques Using greylisting and SpamAssassin

2. The problem The vast majority of email today is Spam Some current statistics indicate over 90% of email Spam This matches my experience

3. Botnets Vast majority of Spam comes from Botnets compromised home PCs hundreds of thousands to millions, or even tens of millions of machines in a heard Controlled by the owner of the heard via a centralised command and control structure Typically don't have a “real” smtp server to actually send the email

4. Spam Reduction with Greylisting and SpamAssassin Currently > 99% effective (closer to 99.8%)‏ In a recent week, only 11 out of 8,000 Spam messages made it through to the end user without being stopped or marked.

5. Spam statistics as of: 16/09/2007 Total spam: 5459 Total greylisted: 4457(90.8%)‏ Total emails accepted (both spam and legitimate): 451 (9.2)% Total identified spam through to end users: 1002 (20.4%)‏ Emails greylist_delayed: 58 (1.2%), marked as spam 57 (96.6%), NOT marked as spam 2 (3.4%)‏ emails via backup mx: 991 (20.2%), marked as spam 944 (95.2%), NOT marked as spam 48 (4.8%)‏ Effectiveness of Greylisting / SpamAssassin: 99.0%. 50 out of 4908 not marked as spam Spam statistics as of: 23/09/2007 Total spam: 5167 Total greylisted: 4928(90.8%)‏ Total emails accepted (both spam and legitimate): 499 (9.2)% Total identified spam through to end users: 239 (4.4%)‏ Emails greylist_delayed: 99 (1.8%), marked as spam 98 (97.0%), NOT marked as spam 3 (3.0%)‏ emails via backup mx: 151 (2.8%), marked as spam 138 (90.2%), NOT marked as spam 15 (9.8%)‏ Effectiveness of Greylisting / SpamAssassin: 99.7%. 18 out of 5427 not marked as spam Spam statistics as of: 30/09/2007 Total spam: 6216 Total greylisted: 5950(91.2%)‏ Total emails accepted (both spam and legitimate): 573 (8.8)% Total identified spam through to end users: 266 (4.1%)‏ Emails greylist_delayed: 141 (2.2%), marked as spam 135 (95.1%), NOT marked as spam 7 (4.9%)‏ emails via backup mx: 151 (2.3%), marked as spam 128 (84.2%), NOT marked as spam 24 (15.8%)‏ Effectiveness of Greylisting / SpamAssassin: 99.5%. 31 out of 6523 not marked as spam Spam statistics as of: 07/10/2007 Total spam: 7901 Total greylisted: 7712(93.0%)‏ Total emails accepted (both spam and legitimate): 581 (7.0)% Total identified spam through to end users: 189 (2.3%)‏ Emails greylist_delayed: 135 (1.6%), marked as spam 134 (97.8%), NOT marked as spam 3 (2.2%)‏ emails via backup mx: 62 (0.7%), marked as spam 55 (87.3%), NOT marked as spam 8 (12.7%)‏ Effectiveness of Greylisting / SpamAssassin: 99.8%. 11 out of 7901 not marked as spam Greylisting removes > 90% of incomming Spam SpamAssassin catches > 90% of received spam Total effectiveness > 99.5%

6. Components Logical layout

7. Greylisting Relies on Spammers not using a “proper” mail server. They just fire-and-forget Give a temporary failure to any “suspect” messages. Spammers will not retry, but a mail server will

8. Which messages to challenge Look at (all of):  From address  To Address  IP of sending machine If not seen before:  give temporary failure  record this “tuple” + time

9. If seen before:  check if it is now past a “start time” (time + time to go live)‏ time to live is typically a parameter passed to greylisting server. many recommend 60 minutes I use 60 seconds  OK – let through record the time  Not OK reject again Any subsequent communication is let straight through

10. Some delay first time someone new contacts you Small chance of non delivery of some messages.  non compliant mail servers  ISPs with rotary pool of mail servers may get continually greylisted  email from web forms that doesn't go through a real mail server Potential issues

11. Risk minimisation Can have various white lists  add mail server details for all regular / potential contacts to a white list these emails are coming from a real mail server, so we don't need to use this test on them. grep you mail server logs to determine who does conatct you. eg: egrep "client=.*mail.*|client=.*mx.*|client=.*smtp.*" /var/log/maillog* | awk '{print $7}' | awk -F = '{print $2}' | awk -F [ '{print $1}' | sort | uniq -u can use regex in these whitelists

12. Examples of server whitelist /^.*\.ebay\.com$/ /.*\.emailebay\.com$/ /^.*\.mx\.bigpond\.com$/ /^.*\.dell\.com\.au$/ /^.*\.mailguard\.com\.au$/ /^mailout.*\.pacific\.net\.au$/ /^mail-out.*\.netspace\.net\.au$/ /^mx.*\.phx\.paypal\.com$/ /^smtp.*\.bis\.ap\.blackberry\.com$/ /^.*\.server-mail\.com$/ /^vscan.*\.westnet\.com\.au$/ /^ihug-mail\.icp-qv1-irony?\.iinet\.net\.au$/

13. Implementations Available for many popular mail servers including MS Exchange

14. SpamAssassin Categorises email as either Spam or Ham (good stuff, not Spam), based on a number of tests Each test may add to the overall score for this email If the total score exceeds a (configurable) limit, it is marked as Spam Highly configurable  personal limits, tests, scoring etc

15. Tests Tests to find words that look like viagra etc Is the sender in a RBL Does the sender match the SPF record  v=spf1 a mx mx:westnet.com.au include:westnet.com.au ~all Does the body look like spam The ratio of text to images Bayesian analysis of the content Many more tests see: http://spamassassin.apache.org/tests_3_2_x.html for the full list

16. Spam / Ham folders can also set up folders containing Spam and Ham (non Spam) for SpamAssassin to learn from. As a large proportion of email is actually spam (if you are not using greylisting), doing this may not be a good idea, as eventually the Bayesian filter gets poisoned and everything ends up looking like spam.

17. Implementations Available for many popular mail servers including MS Exchange  Exchange implementations tend to be commercial offerings

18. SMTP Conversation

19. Greet - Pause When the sender connects, delay the greeting If the sender tries to continue the conversation, before the appropriate response, the conversation is stopped by the smtp server. A “proper” smtp server will handle this, a Spam bot may just have a sequential script and fail this test. About 10% of Spam can be eliminated this way

20. Components (in my system)‏ Postfix mta (postfix-2.3.3-2) http://www.postfix.org postgrey greylisting server (v 1.30) http://postgrey.schweikert.ch/ http://postgrey.schweikert.ch/  See also http://www.greylisting.org/http://www.greylisting.org/ SpamAssassin (spamassassin-3.2.2-1.el5.rf) http://spamassasin.apache.org/ http://spamassasin.apache.org/