How a major ISP built a new anti-abuse platform Mike O’Reirdan Comcast Distinguished Engineer Internet Systems Engineering Comcast National Engineering & Technical Operations

Outline Comcast facts and figures Why build a new platform Fundamentals of anti spam Size of the problem Previous approach Current solution Migration methods Current status

Why a new platform? Moved from a hosted to an in-house platform Need to improve customer experience by further reducing volumes of spam to the mailbox Deploy a platform which can economically and easily scale Emerging threats in abuse landscape Image spam Botnets VoIP spam (SPIT) Need to have a plug-and-play architecture Firmly believe that no one vendor will be the best forever We need a mix of vendors and approaches to hedge our bets and reduce risk Somebody in this room may be our next vendor when you have gone from the lab to the VC and into beta

Size of the problem Volumes of spam are astronomical 596 million connection attempts (Jan25th 2008) 539 million connection attempts rejected 93% spam 76 million messages delivered Connection attempts increases massively above this around holidays such as Thanksgiving. The problems is criminality at massive scale

Fundamentals of anti-spam Not much differentiation between major mail box hosters and other ISPs with regard to spam percentages and volumes Three stages Blocking based on IP (reputation and DUL space) –5% of CPU cycles –Removes ~70% of the spam Blocking based on message protocol and heuristics –10% of CPU cycles –Removes ~15% of the spam Blocking based on content –85% of CPU cycles –Remove ~10% of the spam Idea is to use the least cycles to remove the most messages

Previous approach 100s of Linux blade servers No site fail over Multiple RBLs using BIND for DNS Heuristics and protocol filtering Spam content filtering using industry standard software Virus filtering using industry standard software

New Approach Fewer Linux Blade servers distributed over two sites Full dual site redundancy with each site fully capable of carrying 100% of traffic RBLs hosted on a specialised DNS based platform Trend Spamhaus Return Path Protocol and heuristics filtering performed on the Bizanga IMP MTAs which run on Linux Spam content filtering technology Anti-virus technology

Heuristics employed Directory Harvest attack Dictionary attack rDNS check Throttling Dynamic space blocking Non-existent user block

Content filtering-detecting spammy content Cloudmark Relies on multiple sources of data –Spam / no Spam reports from end users –Honeypots Initially based on Vipul’s Razor Applies algorithmically derived signatures to incoming (Proprietary) Zero hour anti virus Trend Anti-virus Signature analysis Heuristics

Migration Relatively simple process to migrate from old platform Moved traffic across by re-pointing comcast.net MX records to new platform and making lots of involved highly planned DNS configuration changes Performed a series of increasing short duration burst test scale Then moved 5% of the traffic. After platform rules proved stable, traffic was moved across in slightly larger increments over several days to the new platform. This method allowed us to quickly revert back (under 30 minutes) to old platform in the event of any issues without customer impact

Lessons learned It always helps to be able to test the new platform against an existing live flow but this is difficult at our scale with a multi-Gbps mail flow Failing that, heavy reliance has to be placed on cooperation with vendors and existing platform technology users Rules used on an old platform do not always map across neatly to a new one