1. DNS-based Detection of Computer Worms in an Enterprise Environment
David Whyte

2. Outline Internet Environment
(1) Scanning Worm Propagation Characteristics
(2) SMTP-engines
Domain Name System (DNS)-based Detection Approach
Results
Limitations
Conclusions

3. Worm Outbreaks Sapphire/Slammer worm – Jan 25, 2003
Fastest spreading worm yet
90% compromised in first 10 minutes
Doubled in size every 8.5 seconds (first minute)
August 2003 the “Month of Worms”
SoBig.F: #1 “mass mailing” virus of all time
Blaster/LovSan
Welchia/Nachi
Witty worm – March 2004
Buffer overflow in a suite of security products
Use of a hit-list?

4. Top Attacker:
Most Attacked Port: 445

5. DShield Report (May 09, 2005) Top Attacker: 61.152.160.63
Most Attacked Port: 445

6. Countermeasure Challenges
Propagation speed renders human-based defensive strategies non-effective
Security patches – frequent, large and sometimes broken
Slammer fix SQL Server 2000 SP2: three parts 26MB to 506MB
sql2kdeskfullsp2.exe 390 MB

7. Countermeasure Challenges
Active response (i.e. containment and suppression)
risky (self imposed DoS)
The IDS that cried “Worm!!”…
(Snort) alert UDP any any -> any 1434 (msg:"SQL Slammer Worm"; rev:1; content:"|726e51686f756e b |";)
(Dragon) NAME=SMB:DCOM-OVERFLOW SIGNATURE=T D A B SMB:DCOM-OVERFLOW /5c/00/43/00/24/00/5c/00*/00*/00*/00*/00*/00*/00*/00*/00*/00*/00*/00*/00*/00*/00 , /01/10/08/00/cc/cc/cc/cc

8. Solution/Problem
Automated countermeasures are required for worm containment and suppression
Worm propagation detection methods need to:
Detect propagation quickly
Detect propagation accurately
Detect propagation of varying rates
Detect zero-day worms

9. Scanning Worm Characteristics
Scanning worms can employ a variety of strategies to infect systems
Topological scanning
Slow scanning
Fast scanning
Most generate pseudo-random 32-bit numbers to determine their targets
The use of numeric IP addresses does not require a DNS lookup

10. DNS-based Scanning Worm Detection Approach
Enterprise solution
Most legitimate traffic uses the alphanumeric equivalent of an IP address and thus requires a DNS lookup
As the network traffic leaves the network boundary it can easily be determined if a DNS request was involved
If no DNS query is detected for a connection attempt it is considered anomalous

11. Set-up and Training Period
Divide network into cells
Gather DNS requests, embedded IPs in HTTP packets
Construct a candidate connection list (CCL)
Anomaly-based – “training period” required to generate build CCL and whitelist
Detects local to remote (L2R) and local to local (L2L) inter-cell propagation

12. Enterprise Network

13. Cell Monitoring Maintain CCL Monitor outgoing connections (TCP or UDP)
Harvest DNS requests, HTTP (add to CCL)
Respect ttl values (prune CCL)
Monitor outgoing connections (TCP or UDP)
Connection attempts to IPs not found in the CCL or in the whitelist cause an alert

14. Software Developed Prototype uses Perl modules and libpcap
Inline network device
High-level design
Packet Processing Engine (PPE)
Extract features from network traffic
DNS Correlation Engine (DCE)
Create CCL
Maintain whitelist
Validate new connections
Generate alerts

15. Testing Environment Testing on two cells - one week test period
Three hour training period
Cell one: lab network
One quarter Class C network
Small user population
Closed network
Cell two: interdepartmental network (IDN)
Traffic from multiple Class C networks
Large user population
Open network

16. Testing Results (Alerts)
Lab Network
Total alerts: 52
Alerts caused by worm activity: 0
Infected hosts: 0
False Positives: 52
IDN Network
Total alerts:
Alerts caused by worm activity:
Infected hosts: 195
False Positives: 358

17. Detected Worm Propagation
W32.Sasser:
W32.Blaster:
W32.Gaobot:

18. False Positive Analysis
Total false positives: 410
Trojan horse/vulnerability scans: 358
Whitelist activity exclusion: 16
“True” false positives: 36
“True” false positive analysis
Majority caused by low DNS reply ttls coupled with improperly terminating TCP connections

19. False Positive Reduction
Solution
Increase small ttl values to a “reasonable” minimum
Require observation of N anomalous scans within a set time period
With N = 2 only 4 false positives observed
HTTP source of all false positives
Process Host: field in HTTP – 0 false positives

20. DNS-based Worm Detection Advantages
Relies on observation of a protocol found in all networks
Closed network
Detection of zero-day worms / attack tools
Detection of low and slow attacks – no threshold
Low maintenance
Open network
Network traffic filter
Used in conjunction with other worm detection techniques

21. Limitations Will not detect intra-cell propagation Whitelist is static
Open networks cause large whitelists and the potential for false negatives
Will not detect network share traversal propagation or mass mailing worms
Automated scanning/attack tool false positives
P2P traffic!

22. Conclusions Has the potential to detect zero-day worms
Possible worm detection in a single scan no scanning rate constraint (closed network)
Could be used as another detection signal in a more sophisticated tool suite (open network)
Technique could be applied to detect network scanning tools, mass-mailing worms, and covert communications

23. SMTP-engines
SMTP-engine: code that turns a compromised system into a malicious mail server
Used as a propagation mechanism for
Mass-mailing worms: MyDoom, NetSky, Bagle, etc.
Spam: botnets
Mass-mailing worms/spam convergence: SoBig
Phishing: trojans

24. DNS-based Detection Approach
Enterprise solution
Most client systems use an client/corporate mail server to send mail
Determine if a client system performs DNS MX requests
MX requests from non-mail servers is considered anomalous

25. Prototype (proof-of-concept)
Detection
Process MX queries
Whitelist
Containment
IPTables
Configurable alert threshold

26. Testing Environment / Results
University department network
300 client systems
One-week observation period
Only 5 unexplained MX queries observed
Isolated test network
Live mass-mailing worms (NetSky)
Detected and stopped the first malicious mail

27. Uses A replacement for port 25 blocking?
Allows mail relaying
Used in conjunction with Sender Policy Framework (SPF) & DomainKeys
Stops SMTP-engine generated mail before it leaves the network boundary
Natural “add-on” to DNS-based scanning worm detection approach

28. Limitations Does not detect open mail relays
Unable to differentiate between spam relays vs. mass-mailing worm

29. Conclusions A viable alternative to port 25 blocking
Could allow mail relaying
Content-independent detection
Embedded URLs
Has the potential to detect zero-day worms/botnet activity

30. Research Papers
DNS-based Detection of Scanning Worms in an Enterprise Environment, In Proceedings of the 12th Annual Network and Distributed System Security Symposium (NDSS05)
Addressing Malicious SMTP-based Mass-Mailing Activity Within an Enterprise Network **
ARP-based Detection of Scanning Worms Within an Enterprise Environment **
**Technical Report: Carleton University School of Computer Science

31. Questions? .