1. Usenix Security 2004 Autograph Toward Automated, Distributed Worm Signature Detection Hyang-Ah KimBrad Karp Carnegie Mellon UniversityIntel Research & Carnegie Mellon University

2. Usenix Security 20042 Internet Worm Quarantine Internet Worm Quarantine Techniques  Destination port blocking  Infected source host IP blocking  Content-based blocking Worm Signature Content-based blocking [Moore et al., 2003] 05:45:31.912454 90.196.22.196.1716 > 209.78.235.128.80:. 0:1460(1460) ack 1 win 8760 (DF) 0x0000 4500 05dc 84af 4000 6f06 5315 5ac4 16c4E.....@.o.S.Z... 0x0010 d14e eb80 06b4 0050 5e86 fe57 440b 7c3b.N.....P^..WD.|; 0x0020 5010 2238 6c8f 0000 4745 5420 2f64 6566P."8l...GET./def 0x0030 6175 6c74 2e69 6461 3f58 5858 5858 5858ault.ida?XXXXXXX 0x0040 5858 5858 5858 5858 5858 5858 5858 5858XXXXXXXXXXXXXXXX..... 0x00e0 5858 5858 5858 5858 5858 5858 5858 5858XXXXXXXXXXXXXXXX 0x00f0 5858 5858 5858 5858 5858 5858 5858 5858XXXXXXXXXXXXXXXX 0x0100 5858 5858 5858 5858 5858 5858 5858 5858XXXXXXXXXXXXXXXX 0x0110 5858 5858 5858 5858 5825 7539 3039 3025XXXXXXXXX%u9090% 0x01a0 303d 6120 4854 5450 2f31 2e30 0d0a 436f0=a.HTTP/1.0..Co. Signature for CodeRed II Signature : A Payload Content String Specific To A Worm

3. Usenix Security 20043 Content-based Blocking Our network X Traffic Filtering Internet Signature for CodeRed II  Can be used by Bro, Snort, Cisco’s NBAR,...

4. Usenix Security 20044 Signature derivation is too slow Current Signature Derivation Process  New worm outbreak  Report of anomalies from people via phone/email/newsgroup  Worm trace is captured  Manual analysis by security experts  Signature generation  Labor-intensive, Human-mediated

5. Usenix Security 20045 Goal Automatically generate signatures of previously unknown Internet worms  as accurately as possible  as quickly as possible  Content-Based Analysis  Automation, Distributed Monitoring

6. Usenix Security 20046 Assumptions We focus on TCP worms that propagate via scanning Actually, any transport  in which spoofed sources cannot communicate successfully  in which transport framing is known to monitor Worm’s payloads share a common substring  Vulnerability exploit part is not easily mutable Not polymorphic

7. Usenix Security 20047 Outline Problem and Motivation Automated Signature Detection  Desiderata  Technique  Evaluation Distributed Signature Detection  Tattler  Evaluation Related Work Conclusion

8. Usenix Security 20048 Desiderata Automation: Minimal manual intervention Signature quality: Sensitive & specific  Sensitive: match all worms  low false negative rate  Specific: match only worms  low false positive rate Timeliness: Early detection Application neutrality  Broad applicability

9. Usenix Security 20049 Automated Signature Generation Step 1: Select suspicious flows using heuristics Step 2: Generate signature using content- prevalence analysis Our network Traffic Filtering Internet Autograph Monitor Signature X

10. Usenix Security 200410 Heuristic: Flows from scanners are suspicious  Focus on the successful flows from IPs who made unsuccessful connections to more than s destinations for last 24hours  Suitable heuristic for TCP worm that scans network Suspicious Flow Pool  Holds reassembled, suspicious flows captured during the last time period t  Triggers signature generation if there are more than  flows S1: Suspicious Flow Selection Reduce the work by filtering out vast amount of innocuous flows Autograph (s = 2) Non-existent  This flow will be selected

11. Usenix Security 200411 S1: Suspicious Flow Selection Heuristic: Flows from scanners are suspicious  Focus on the successful flows from IPs who made unsuccessful connections to more than s destinations for last 24hours  Suitable heuristic for TCP worm that scans network Suspicious Flow Pool  Holds reassembled, suspicious flows captured during the last time period t  Triggers signature generation if there are more than  flows Reduce the work by filtering out vast amount of innocuous flows

12. Usenix Security 200412 S2: Signature Generation All instances of a worm have a common byte pattern specific to the worm Rationale  Worms propagate by duplicating themselves  Worms propagate using vulnerability of a service Use the most frequent byte sequences across suspicious flows as signatures How to find the most frequent byte sequences?

13. Usenix Security 200413 Worm-specific Pattern Detection Use the entire payload  Brittle to byte insertion, deletion, reordering GARBAGEEABCDEFGHIJKABCDXXXX Flow 1 Flow 2 GARBAGEABCDEFGHIJKABCDXXXXX

14. Usenix Security 200414 Worm-specific Pattern Detection Partition flows into non-overlapping small blocks and count the number of occurrences Fixed-length Partition  Still brittle to byte insertion, deletion, reordering GARBAGEEABCDEFGHIJKABCDXXXX Flow 1 Flow 2 GARBAGEABCDEFGHIJKABCDXXXXX

15. Usenix Security 200415 Worm-specific Pattern Detection Content-based Payload Partitioning (COPP)  Partition if Rabin fingerprint of a sliding window matches Breakmark  Configurable parameters: content block size (minimum, average, maximum), breakmark, sliding window  Content Blocks Breakmark = last 8 bits of fingerprint ( ABCD ) GARBAGEEABCDEFGHIJKABCDXXXX Flow 1 Flow 2 GARBAGEABCDEFGHIJKABCDXXXXX

16. Usenix Security 200416 Why Prevalence? Worm flows dominate in the suspicious flow pool Content-blocks from worms are highly ranked Nimda CodeRed2 Nimda (16 different payloads) WebDAV exploit Innocuous, misclassified Prevalence Distribution in Suspicious Flow Pool - From 24-hr http traffic trace

17. Usenix Security 200417 Select Most Frequent Content Block ABD ABE ACE AD CF CDG B f0 f1 f2 f3 f4 f5 HIJ f6 IHJ f7 GIJ f8

18. Usenix Security 200418 A A A E E A F C C C D D DB B B H H G G I I I J J J Select Most Frequent Content Block D C E E A A A A D F C C DG B B B H H G I I I J J J f0 f1 f2 f3 f4 f5 f6 f7 f8 f0 C F f1 C D G f2 A B D f3 A C E f4 A B E f5 A B D f6 H I J f7 I H J f8 G I J

19. Usenix Security 200419 Select Most Frequent Content Block A B D A BE A C E A D C F C D G B H I J I H J GI J f0 C F f1 C D G f2 A B D f3 A C E f4 A B E f5 A B D f6 H I J f7 I H J f8 G I J P≥3P≥3 W ≥ 90% Signature: W: target coverage in suspicious flow pool P: minimum occurrence to be selected

20. Usenix Security 200420 Signature: A Select Most Frequent Content Block A B D A BE A C E A D C F C D G B H I J I H J GI J f0 C F f1 C D G f2 A B D f3 A C E f4 A B E f5 A B D f6 H I J f7 I H J f8 G I J P≥3P≥3 W ≥ 90% W: target coverage in suspicious flow pool P: minimum occurrence to be selected

21. Usenix Security 200421 Select Most Frequent Content Block B DB A A A C E E A D F C C D G B H I J I H J GI J P≥3P≥3 W ≥ 90% Signature: A f0 C F f1 C D G f2 A B D f3 A C E f4 A B E f5 A B D f6 H I J f7 I H J f8 G I J W: target coverage in suspicious flow pool P: minimum occurrence to be selected

22. Usenix Security 200422 Select Most Frequent Content Block F C C D G H I J I H J GI J P≥3P≥3 W ≥ 90% Signature: A f0 C F f1 C D G f2 A B D f3 A C E f4 A B E f5 A B D f6 H I J f7 I H J f8 G I J I W: target coverage in suspicious flow pool P: minimum occurrence to be selected

23. Usenix Security 200423 Select Most Frequent Content Block F C C DG P≥3P≥3 W ≥ 90% Signature: A f0 C F f1 C D G f2 A B D f3 A C E f4 A B E f5 A B D f6 H I J f7 I H J f8 G I J I Signature: W: target coverage in suspicious flow pool P: minimum occurrence to be selected

24. Usenix Security 200424 Outline Problem and Motivation Automated Signature Detection  Desiderata  Technique  Evaluation Distributed Signature Detection  Tattler  Evaluation Related Work Conclusion

25. Usenix Security 200425 Behavior of Signature Generation Objectives  Effect of COPP parameters on signature quality Metrics  Sensitivity = # of true alarms / total # of worm flows  false negatives  Efficiency = # of true alarms / # of alarms  false positives Trace  Contains 24-hour http traffic  Includes 17 different types of worm payloads

26. Usenix Security 200426 Signature Quality Larger block sizes generate more specific signatures A range of w (90-95%, workload dependent) produces a good signature

27. Usenix Security 200427 Outline Problem and Motivation Automated Signature Detection  Desiderata  Technique  Evaluation Distributed Signature Detection  Tattler  Evaluation Related Work Conclusion

28. Usenix Security 200428 Signature Generation Speed Bounded by worm payload accumulation speed  Aggressiveness of scanner detection heuristic s: # of failed connection peers to detect a scanner  # of payloads enough for content analysis  : suspicious flow pool size to trigger signature generation Single Autograph  Worm payload accumulation is slow Internet AAAAAAA tattler Distributed Autograph  Share scanner IP list  Tattler: limit bandwidth consumption within a predefined cap

29. Usenix Security 200429 Benefit from tattler Worm payload accumulation (time to catch 5 worms) Signature generation  More aggressive scanner detection (s) and signature generation trigger (  )  faster signature generation, more false positives  With s=2 and  =15, Autograph generates the good worm signature before < 2% hosts get infected Info Sharing Autograph Monitor Fraction of Infected Hosts Aggressive (s = 1) Conservative (s = 4) None Luckiest2%60% Median25% -- TattlerAll<1%15% Many innocuous misclassified flows

30. Usenix Security 200430 Related Work Automated Worm Signature Detection Distributed Monitoring  Honeyd [Provos2003], DOMINO [Yegneswaran et al. 2004]  Corroborate faster accumulation of worm payloads/scanner IPs EarlyBird [Singh et al. 2003] HoneyComb [Kreibich et al. 2003] Autograph Signature Generation Content prevalence  Address Dispersion Honeypot + Pairwise LCS Suspicious flow selection  Content prevalence DeploymentNetworkHostNetwork Flow Reassembly NoYes Distributed Monitoring No Yes

31. Usenix Security 200431 Future Work Attacks  Overload Autograph  Abuse Autograph for DoS attacks Online evaluation with diverse traces & deployment on distributed sites Broader set of suspicious flow selection heuristics  Non-scanning worms (ex. hit-list worms, topological worms, email worms)  UDP worms Egress detection Distributed agreement for signature quality testing  Trusted aggregation

32. Usenix Security 200432 Conclusion Stopping spread of novel worms requires early generation of signatures Autograph: automated signature detection system  Automated suspicious flow selection → Automated content prevalence analysis  COPP: robustness against payload variability  Distributed monitoring: faster signature generation Autograph finds sensitive & specific signatures early in real network traces

33. Usenix Security 2004 For more information, visit http://www.cs.cmu.edu/~hakim/autograph

34. Usenix Security 200434 Attacks Overload due to flow reassembly Solutions  Multiple instances of Autograph on separate HW (port-disjoint)  Suspicious flow sampling under heavy load Abuse Autograph for DoS: pollute suspicious flow pool  Port scan and then send innocuous traffic Solution  Distributed verification of signatures at many monitors  Source-address-spoofed port scan Solution  Reply with SYN/ACK on behalf of non-existent hosts/services

35. Usenix Security 200435 Number of Signatures Smaller block sizes generate small # of signatures

36. Usenix Security 200436 tattler A modified RTCP (RTP Control Protocol) Limit the total bandwidth of announcements sent to the group within a predetermined cap

37. Usenix Security 200437 Simulation Setup About 340,000 vulnerable hosts from about 6400 ASes Took small size edge networks (/16s) based on BGP table of 19 th of July, 2001. Service deployment  50% of address space within the vulnerable ASes is reachable  25% of reachable hosts run web server  340,000 vulnerable hosts are randomly placed. Scanning  10probes per second  Scanning the entire non-class-D IP address space Network/processing delays  Randomly chosen in [0.5, 1.5] seconds