Spam Reduction Techniques Using greylisting and SpamAssassin

The problem The vast majority of today is Spam Some current statistics indicate over 90% of Spam This matches my experience

Botnets Vast majority of Spam comes from Botnets compromised home PCs hundreds of thousands to millions, or even tens of millions of machines in a heard Controlled by the owner of the heard via a centralised command and control structure Typically don't have a “real” smtp server to actually send the

Spam Reduction with Greylisting and SpamAssassin Currently > 99% effective (closer to 99.8%)‏ In a recent week, only 11 out of 8,000 Spam messages made it through to the end user without being stopped or marked.

Spam statistics as of: 16/09/2007 Total spam: 5459 Total greylisted: 4457(90.8%)‏ Total s accepted (both spam and legitimate): 451 (9.2)% Total identified spam through to end users: 1002 (20.4%)‏ s greylist_delayed: 58 (1.2%), marked as spam 57 (96.6%), NOT marked as spam 2 (3.4%)‏ s via backup mx: 991 (20.2%), marked as spam 944 (95.2%), NOT marked as spam 48 (4.8%)‏ Effectiveness of Greylisting / SpamAssassin: 99.0%. 50 out of 4908 not marked as spam Spam statistics as of: 23/09/2007 Total spam: 5167 Total greylisted: 4928(90.8%)‏ Total s accepted (both spam and legitimate): 499 (9.2)% Total identified spam through to end users: 239 (4.4%)‏ s greylist_delayed: 99 (1.8%), marked as spam 98 (97.0%), NOT marked as spam 3 (3.0%)‏ s via backup mx: 151 (2.8%), marked as spam 138 (90.2%), NOT marked as spam 15 (9.8%)‏ Effectiveness of Greylisting / SpamAssassin: 99.7%. 18 out of 5427 not marked as spam Spam statistics as of: 30/09/2007 Total spam: 6216 Total greylisted: 5950(91.2%)‏ Total s accepted (both spam and legitimate): 573 (8.8)% Total identified spam through to end users: 266 (4.1%)‏ s greylist_delayed: 141 (2.2%), marked as spam 135 (95.1%), NOT marked as spam 7 (4.9%)‏ s via backup mx: 151 (2.3%), marked as spam 128 (84.2%), NOT marked as spam 24 (15.8%)‏ Effectiveness of Greylisting / SpamAssassin: 99.5%. 31 out of 6523 not marked as spam Spam statistics as of: 07/10/2007 Total spam: 7901 Total greylisted: 7712(93.0%)‏ Total s accepted (both spam and legitimate): 581 (7.0)% Total identified spam through to end users: 189 (2.3%)‏ s greylist_delayed: 135 (1.6%), marked as spam 134 (97.8%), NOT marked as spam 3 (2.2%)‏ s via backup mx: 62 (0.7%), marked as spam 55 (87.3%), NOT marked as spam 8 (12.7%)‏ Effectiveness of Greylisting / SpamAssassin: 99.8%. 11 out of 7901 not marked as spam Greylisting removes > 90% of incomming Spam SpamAssassin catches > 90% of received spam Total effectiveness > 99.5%

Components Logical layout

Greylisting Relies on Spammers not using a “proper” mail server. They just fire-and-forget Give a temporary failure to any “suspect” messages. Spammers will not retry, but a mail server will

Which messages to challenge Look at (all of):  From address  To Address  IP of sending machine If not seen before:  give temporary failure  record this “tuple” + time

If seen before:  check if it is now past a “start time” (time + time to go live)‏ time to live is typically a parameter passed to greylisting server. many recommend 60 minutes I use 60 seconds  OK – let through record the time  Not OK reject again Any subsequent communication is let straight through

Some delay first time someone new contacts you Small chance of non delivery of some messages.  non compliant mail servers  ISPs with rotary pool of mail servers may get continually greylisted  from web forms that doesn't go through a real mail server Potential issues

Risk minimisation Can have various white lists  add mail server details for all regular / potential contacts to a white list these s are coming from a real mail server, so we don't need to use this test on them. grep you mail server logs to determine who does conatct you. eg: egrep "client=.*mail.*|client=.*mx.*|client=.*smtp.*" /var/log/maillog* | awk '{print $7}' | awk -F = '{print $2}' | awk -F [ '{print $1}' | sort | uniq -u can use regex in these whitelists

Examples of server whitelist /^.*\.ebay\.com$/ /.*\. ebay\.com$/ /^.*\.mx\.bigpond\.com$/ /^.*\.dell\.com\.au$/ /^.*\.mailguard\.com\.au$/ /^mailout.*\.pacific\.net\.au$/ /^mail-out.*\.netspace\.net\.au$/ /^mx.*\.phx\.paypal\.com$/ /^smtp.*\.bis\.ap\.blackberry\.com$/ /^.*\.server-mail\.com$/ /^vscan.*\.westnet\.com\.au$/ /^ihug-mail\.icp-qv1-irony?\.iinet\.net\.au$/

Implementations Available for many popular mail servers including MS Exchange

SpamAssassin Categorises as either Spam or Ham (good stuff, not Spam), based on a number of tests Each test may add to the overall score for this If the total score exceeds a (configurable) limit, it is marked as Spam Highly configurable  personal limits, tests, scoring etc

Tests Tests to find words that look like viagra etc Is the sender in a RBL Does the sender match the SPF record  v=spf1 a mx mx:westnet.com.au include:westnet.com.au ~all Does the body look like spam The ratio of text to images Bayesian analysis of the content Many more tests see: for the full list

Spam / Ham folders can also set up folders containing Spam and Ham (non Spam) for SpamAssassin to learn from. As a large proportion of is actually spam (if you are not using greylisting), doing this may not be a good idea, as eventually the Bayesian filter gets poisoned and everything ends up looking like spam.

Implementations Available for many popular mail servers including MS Exchange  Exchange implementations tend to be commercial offerings

SMTP Conversation

Greet - Pause When the sender connects, delay the greeting If the sender tries to continue the conversation, before the appropriate response, the conversation is stopped by the smtp server. A “proper” smtp server will handle this, a Spam bot may just have a sequential script and fail this test. About 10% of Spam can be eliminated this way

Components (in my system)‏ Postfix mta (postfix ) postgrey greylisting server (v 1.30)  See also SpamAssassin (spamassassin el5.rf)