DNS-based Detection of Computer Worms in an Enterprise Environment David Whyte

Outline Internet Environment (1) Scanning Worm Propagation Characteristics (2) SMTP-engines Domain Name System (DNS)-based Detection Approach Results Limitations Conclusions

Worm Outbreaks Sapphire/Slammer worm – Jan 25, 2003 Fastest spreading worm yet 90% compromised in first 10 minutes Doubled in size every 8.5 seconds (first minute) August 2003 the “Month of Worms” SoBig.F: #1 “mass mailing” virus of all time Blaster/LovSan Welchia/Nachi Witty worm – March 2004 Buffer overflow in a suite of security products Use of a hit-list?

Top Attacker: 61.152.160.63 Most Attacked Port: 445

DShield Report (May 09, 2005) Top Attacker: 61.152.160.63 Most Attacked Port: 445

Countermeasure Challenges Propagation speed renders human-based defensive strategies non-effective Security patches – frequent, large and sometimes broken Slammer fix SQL Server 2000 SP2: three parts 26MB to 506MB sql2kdeskfullsp2.exe 390 MB

Countermeasure Challenges Active response (i.e. containment and suppression) risky (self imposed DoS) The IDS that cried “Worm!!”… (Snort) alert UDP any any -> any 1434 (msg:"SQL Slammer Worm"; rev:1; content:"|726e51686f756e746869636b43684765|";) (Dragon) NAME=SMB:DCOM-OVERFLOW SIGNATURE=T D A B 2 0 135 SMB:DCOM-OVERFLOW /5c/00/43/00/24/00/5c/00*/00*/00*/00*/00*/00*/00*/00*/00*/00*/00*/00*/00*/00*/00 , /01/10/08/00/cc/cc/cc/cc

Solution/Problem Automated countermeasures are required for worm containment and suppression Worm propagation detection methods need to: Detect propagation quickly Detect propagation accurately Detect propagation of varying rates Detect zero-day worms

Scanning Worm Characteristics Scanning worms can employ a variety of strategies to infect systems Topological scanning Slow scanning Fast scanning Most generate pseudo-random 32-bit numbers to determine their targets The use of numeric IP addresses does not require a DNS lookup

DNS-based Scanning Worm Detection Approach Enterprise solution Most legitimate traffic uses the alphanumeric equivalent of an IP address and thus requires a DNS lookup As the network traffic leaves the network boundary it can easily be determined if a DNS request was involved If no DNS query is detected for a connection attempt it is considered anomalous

Set-up and Training Period Divide network into cells Gather DNS requests, embedded IPs in HTTP packets Construct a candidate connection list (CCL) Anomaly-based – “training period” required to generate build CCL and whitelist Detects local to remote (L2R) and local to local (L2L) inter-cell propagation

Enterprise Network

Cell Monitoring Maintain CCL Monitor outgoing connections (TCP or UDP) Harvest DNS requests, HTTP (add to CCL) Respect ttl values (prune CCL) Monitor outgoing connections (TCP or UDP) Connection attempts to IPs not found in the CCL or in the whitelist cause an alert

Software Developed Prototype uses Perl modules and libpcap Inline network device High-level design Packet Processing Engine (PPE) Extract features from network traffic DNS Correlation Engine (DCE) Create CCL Maintain whitelist Validate new connections Generate alerts

Testing Environment Testing on two cells - one week test period Three hour training period Cell one: lab network One quarter Class C network Small user population Closed network Cell two: interdepartmental network (IDN) Traffic from multiple Class C networks Large user population Open network

Testing Results (Alerts) Lab Network Total alerts: 52 Alerts caused by worm activity: 0 Infected hosts: 0 False Positives: 52 IDN Network Total alerts: 74 968 Alerts caused by worm activity: 74 610 Infected hosts: 195 False Positives: 358

Detected Worm Propagation W32.Sasser: 49 425 W32.Blaster: 7 014 W32.Gaobot: 18 171

False Positive Analysis Total false positives: 410 Trojan horse/vulnerability scans: 358 Whitelist activity exclusion: 16 “True” false positives: 36 “True” false positive analysis Majority caused by low DNS reply ttls coupled with improperly terminating TCP connections

False Positive Reduction Solution Increase small ttl values to a “reasonable” minimum Require observation of N anomalous scans within a set time period With N = 2 only 4 false positives observed HTTP source of all false positives Process Host: field in HTTP – 0 false positives

DNS-based Worm Detection Advantages Relies on observation of a protocol found in all networks Closed network Detection of zero-day worms / attack tools Detection of low and slow attacks – no threshold Low maintenance Open network Network traffic filter Used in conjunction with other worm detection techniques

Limitations Will not detect intra-cell propagation Whitelist is static Open networks cause large whitelists and the potential for false negatives Will not detect network share traversal propagation or mass mailing worms Automated scanning/attack tool false positives P2P traffic!

Conclusions Has the potential to detect zero-day worms Possible worm detection in a single scan no scanning rate constraint (closed network) Could be used as another detection signal in a more sophisticated tool suite (open network) Technique could be applied to detect network scanning tools, mass-mailing worms, and covert communications

SMTP-engines SMTP-engine: code that turns a compromised system into a malicious mail server Used as a propagation mechanism for Mass-mailing worms: MyDoom, NetSky, Bagle, etc. Spam: botnets Mass-mailing worms/spam convergence: SoBig Phishing: trojans

DNS-based Detection Approach Enterprise solution Most client systems use an email client/corporate mail server to send mail Determine if a client system performs DNS MX requests MX requests from non-mail servers is considered anomalous

Prototype (proof-of-concept) Detection Process MX queries Whitelist Containment IPTables Configurable alert threshold

Testing Environment / Results University department network 300 client systems One-week observation period Only 5 unexplained MX queries observed Isolated test network Live mass-mailing worms (NetSky) Detected and stopped the first malicious mail

Uses A replacement for port 25 blocking? Allows mail relaying Used in conjunction with Sender Policy Framework (SPF) & DomainKeys Stops SMTP-engine generated mail before it leaves the network boundary Natural “add-on” to DNS-based scanning worm detection approach

Limitations Does not detect open mail relays Unable to differentiate between spam relays vs. mass-mailing worm

Conclusions A viable alternative to port 25 blocking Could allow mail relaying Content-independent detection Embedded URLs Has the potential to detect zero-day worms/botnet activity

Research Papers DNS-based Detection of Scanning Worms in an Enterprise Environment, In Proceedings of the 12th Annual Network and Distributed System Security Symposium (NDSS05) Addressing Malicious SMTP-based Mass-Mailing Activity Within an Enterprise Network ** ARP-based Detection of Scanning Worms Within an Enterprise Environment ** **Technical Report: Carleton University School of Computer Science

Questions? dlwhyte@scs.carleton.ca .