1. May 23, 20031 Filtering Emails for Viruses and Spam at DESY Wolfgang Friebel

2. May 23, 20031 Contents ● Background information: size of the problem ● Virus filtering ● Spam tagging ● Evaluation of different tools ● Present status of filtering at DESY

3. May 23, 20031 Spam mail statistics ● Mails received at Zeuthen in 2003

4. May 23, 20031 Spam mail statistics (2) ● Spam Mails I received since Jan 2000 ● 1 day/year lost assuming 50 spams/day at 3s/spam

5. May 23, 20031 Virus mail statistics ● Number of quarantined mails at DESY in the last month (from approx 20-30k mails/day)

6. May 23, 20031 Filtering mail for Viruses ● Problems to be solved – Keep virus signatures up to date – Handle quarantined mail properly – Find viruses even in nested archives – Well behaving servers under high load – Opting out desirable (UNIX users)

7. May 23, 20031 Tools for finding viruses in email at DESY ● Two different approaches were tried – Integrated commercial solution: Mimesweeper (Hamburg) using F-Prot Scanner – Commercial scanner (McAfee) within open source tool amavisd (Zeuthen) ● Mimesweeper in production (Hamburg) – Very good at finding viruses within nested archives – Users get notified of quarantined email, will be deleted after notification (kept 30 days) – Load distributed among 3 machines

8. May 23, 20031 Tools for finding viruses (2) ● amavisd/McAfee evaluated, currently not used (Zeuthen) – Windows computers at Zeuthen are managed centrally and do have running virus scanners – Filtering for viruses would generate additional load on the mail server which is close to its limit – Additional security comes at a high price – Will definitely give it another try when users migrated to new mail server, then opt in/opt out using amavisd is envisaged

9. May 23, 20031 Identifying spam mails ● Mail tagging – Mails from other sites get tagged (Zeuthen: all mails) – Only for mails < 250 kbytes – Product used: Spamassassin – additional mechanisms provided by Mimesweeper in HH ● No mail filtering – No mails will be thrown away – Decision to filter is left to the user – Several mechanisms (see later)

10. May 23, 20031 Mail tagging ● Still trying to find the optimum solution: [SPAM] in the Subject: line (Hamburg) – good visibility, easy filtering, problems when forwarding mail misclassified as spam X-Spam-Level: extra header line (Zeuthen) – normally not visible (use e.g. roles in pine), more fine grained control for filtering, forwarding is ok Altering the mail body (Hamburg) – Again good for visibility, but changes content (bad for filtering tools at other sites)

11. May 23, 20031 Interaction with the MTA ● Different solutions for different MTA's – MTA usually cannot call spamassassin directly – A call to spamassassin is starting perl ● Multithreading daemon prevents forking perl ● For sendmail the milter interface is used – miltrassassin as glue between sendmail and spamd – mime-defang is a milter and calls spamassassin directly, no need to use spamd, used for virus filtering as well

12. May 23, 20031 Interaction with the MTA (2) ● Postfix can use filters (modifying the email) – amavisd is very powerful and flexible, handles also virus scanners, allows for opt in/out, when used with sendmail no mail tagging possible ● Solutions for other MTA's exist (qmail, exim, Exchange) but were not looked at ● Zeuthen: sendmail+miltrassassin+spamd ● Hamburg: Mimesweeper (calls spamassassin) ● Both sites plan to use postfix+amavisd in the near future

13. May 23, 20031 Results of the spam tagging ● Concern: good mails tagged as spam (false positives) ● spamassassin improved a lot since Nov '02 ● Rate of false positives decreased after tuning – enabling network tests within spamassassin – switching on bayes filters and autolearning – Whitelisting in pathological cases ● Rate of false positives in Zeuthen << 1:10 000 (1 mail with score 5.0 reported during last two months) ● Rate of false positives in Hamburg higher (less tuning)

14. May 23, 20031 False positives Did you receive good mails in the SPAM folder recently? I'm very happy with the SPAM filter, I haven't seen one false positive ! das ist nicht der Fall. 433 spam emails, keine davon missinterpretiert No. The spam filter works well. Seit März noch nie eine fehlgeleitete "gute" mail bei mir war bis jetzt keine einzige 'gute' Mail im SPAM-Ordner. Approximately zero Not a single godd mail! No trace of ham in my spambox. seit dem 25.4. hatte ich keine gute mail im Spam Folder. In der ganzen Zeit ist nicht eine gute mail im SPAM Folder gelandet bei mir war aller Inhalt ausnahmslos wirklicher spam

15. May 23, 20031 Bayes filtering in spamassassin ● Spamassassin has a so called bayes filter implemented – Based on the frequency of words within good mails vs. frequency of words within bad mails – Calculates a probability for mail being spam ● Autolearning assumes, that all mails below/above a certain score are good/bad mails (we are using -5/+10) – Does already a good job – Help it by sending misclassified mails with all headers to special email addresses (will be processed in a cron job)

16. May 23, 20031 Tagging statistics ● At score=5 roughly 5 percent spam in good mails, no good mail with score > 5

17. May 23, 20031 Filtering Spam mails ● Two choices: – Let the mail server (calling procmail) do the work and have a spam folder besides the INBOX on the server – Do the filtering in the mail reader, i.e. set up a filtering rule The second option is preferred (less labour intensive for admins) ● Recipes on DESY web pages describe how to set up filters for pine, netscape and outlook

18. May 23, 20031 Precautions against spammers ● Open the LDAP port to selected sites only ● The LDAP servers at HEP sites are being abused! ● No personal email URL's on web sites ● But a picture showing it is safe ● Close security holes in web browsers ● Close protocols like identd to the outside ● Avoid “free” services where you have to register by email

19. May 23, 20031 Next steps ● Upgrade spamassassin to latest version ● Zeuthen is using 2.53, latest is 2.54 ● Weighting of tests adapted to patterns of spammers ● Use more network tests ● Since Apr 26 many RBL's included ● Since May 15 razor2 included ● First numbers suggest 97.5 percent suppression at score level 5 (recommended by us) ● Reject incoming email with a higher score level (e.g. 8) ● Already at MTA level, similat to e.g. 'user unknown'

20. May 23, 20031 Conclusion ● Virus filtering well established (Hamburg only) ● Spam tagging still somewhat experimental ● Currently recognition of spam is at the 95 percent level with an extremely low error rate (false positives) ● Users are very positive about the implemented methods ● No central mail filtering is done (but under discussion) – Users need to set up filters to let filtering take place – Still too much responsibility left to users – need to respect the strict german laws