1. Automated Worm Fingerprinting Sumeet Singh, Cristian Estan, George Varghese, and Stefan Savage

2. Introduction Problem: how to react quickly to worms? CodeRed 2001  Infected ~360,000 hosts within 11 hours Sapphire/Slammer (376 bytes) 2002  Infected ~75,000 hosts within 10 minutes

3. Existing Approaches Detection  Ad hoc intrusion detection Characterization  Manual signature extraction Isolates and decompiles a new worm Look for and test unique signatures Can take hours or days

4. Existing Approaches Containment  Updates to anti-virus and network filtering products

5. Earlybird Automatically detect and contain new worms Two observations  Some portion of the content in existing worms is invariant  Rare to see the same string recurring from many sources to many destinations

6. Earlybird Automatically extract the signature of all known worms  Also Blaster, MyDoom, and Kibuv.B hours or days before any public signatures were distributed Few false positives

7. Background and Related Work Almost all IPs were scanned by Slammer < 10 minutes  Limited only by bandwidth constraints

8. The SQL Slammer Worm: 30 Minutes After “Release” - Infections doubled every 8.5 seconds - Spread 100X faster than Code Red - At peak, scanned 55 million hosts per second.

9. Network Effects Of The SQL Slammer Worm At the height of infections  Several ISPs noted significant bandwidth consumption at peering points  Average packet loss approached 20%  South Korea lost almost all Internet service for period of time  Financial ATMs were affected  Some airline ticketing systems overwhelmed

10. Signature-Based Methods Pretty effective if signatures can be generated quickly  For CodeRed, 60 minutes  For Slammer, 1 – 5 minutes

11. Worm Detection Three classes of methods  Scan detection  Honeypots  Behavioral techniques

12. Scan Detection Look for unusual frequency and distribution of address scanning Limitations  Not suited to worms that spread in a non- random fashion (i.e. emails, IM, P2P apps) Based on a target list Spread topologically

13. Scan Detection More limitations  Detects infected sites  Does not produce a signature

14. Honeypots Monitored idle hosts with untreated vulnerabilities  Used to isolate worms Limitations  Manual extraction of signatures  Depend on quick infections

15. Behavioral Detection Looks for unusual system call patterns  Sending a packet from the same buffer containing a received packet  Can detect slow moving worms Limitations  Needs application-specific knowledge  Cannot infer a large-scale outbreak

16. Characterization Process of analyzing and identifying a new worm Current approaches  Use a priori vulnerability signatures  Automated signature extraction

17. Vulnerability Signatures Example  Slammer Worm UDP traffic on port 1434 that is longer than 100 bytes (buffer overflow) Can be deployed before the outbreak  Can only be applied to well-known vulnerabilities

18. Some Automated Signature Extraction Techniques Allows viruses to infect decoy programs  Extracts the modified regions of the decoy  Uses heuristics to identify invariant code strings across infected instances

19. Some Automated Signature Extraction Techniques Limitation  Assumes the presence of a virus in a controlled environment

20. Some Automated Signature Extraction Techniques Honeycomb  Find longest common subsequences among sets of strings found in messages Autograph  Uses network-level data to infer worm signatures Limitations  Scale and full distributed deployments

21. Containment Mechanism used to deter the spread of an active worm  Host quarantine Via IP ACLs on routers or firewalls  String-matching  Connection throttling On all outgoing connections

22. Host Quarantine Preventing an infected host from talking to other hosts  Via IP ACLs on routers or firewalls

23. Defining Worm Behavior Content invariance  Portions of a worm are invariant (e.g. the decryption routine) Content prevalence  Appears frequently on the network Address dispersion  Distribution of destination addresses more uniform to spread fast

24. Finding Worm Signatures Traffic pattern is sufficient for detecting worms  Relatively straightforward  Extract all possible substrings  Raise an alarm when FrequencyCounter[substring] > threshold1 SourceCounter[substring] > threshold2 DestCounter[substring] > threshold3

25. Practical Content Sifting Characteristics  Small processing requirements  Small memory requirements  Allows arbitrary deployment strategies

26. Estimating Content Prevalence Finding the packet payloads that appear at least x times among the N packets sent  During a given interval

27. Estimating Content Prevalence Table[payload]  1 GB table filled in 10 seconds Table[hash[payload]]  1 GB table filled in 4 minutes  Tracking millions of ants to track a few elephants  Collisions...false positives

28. [Singh et al. 2002] stream memory Array of counters Hash(Pink) Multistage Filters

29. packet memory Array of counters Hash(Green) Multistage Filters

30. packet memory Array of counters Hash(Green) Multistage Filters

31. packet memory Multistage Filters

32. packet memory Collisions are OK Multistage Filters

33. packet memory packet1 1 Insert Reached threshold Multistage Filters

34. packet memory packet1 1 Multistage Filters

35. packet memory packet1 1 packet2 1 Multistage Filters

36. Stage 2 packet memory packet1 1 Stage 1 Multistage Filters No false negatives! (guaranteed detection)

37. Gray = all prior packets Conservative Updates

38. Redundant Conservative Updates

40. Detecting Common Strings Cannot afford to detect all substrings Maybe can afford to detect all strings with a small fixed length

41. Detecting Common Strings Cannot afford to detect all substrings Maybe can afford to detect all strings with a small fixed length A horse is a horse, of course, of course F 1 = (c 1 p 4 + c 2 p 3 + c 3 p 2 + c 4 p 1 + c 5 ) mod M

42. Detecting Common Strings Cannot afford to detect all substrings Maybe can afford to detect all strings with a small fixed length A horse is a horse, of course, of course F 1 = (c 1 p 4 + c 2 p 3 + c 3 p 2 + c 4 p 1 + c 5 ) mod M F 2 = (c 2 p 4 + c 3 p 3 + c 4 p 2 + c 5 p 1 + c 6 ) mod M

43. Detecting Common Strings Cannot afford to detect all substrings Maybe can afford to detect all strings with a small fixed length A horse is a horse, of course, of course F 1 = (c 1 p 4 + c 2 p 3 + c 3 p 2 + c 4 p 1 + c 5 ) mod M F 2 = (c 2 p 4 + c 3 p 3 + c 4 p 2 + c 5 p 1 + c 6 ) mod M = (c 1 p 5 + c 2 p 4 + c 3 p 3 + c 4 p 2 + c 5 p 1 + c 6 - c 1 p 5 ) mod M = (pF 1 + c 6 - c 1 p 5 ) mod M

44. Detecting Common Strings Cannot afford to detect all substrings Maybe can afford to detect all strings with a small fixed length  Still too expensive…

45. Estimating Address Dispersion Not sufficient to count the number of source and destination pairs  e.g. send a mail to a mailing list Two sources—mail server and the sender Many destinations Need to count the unique source and destination traffic flows  For each substring

46. [Estan et al. 2003] Bitmap counting – direct bitmap HASH(green)=10001001 Set bits in the bitmap using hash of the flow ID of incoming packets

47. Bitmap counting – direct bitmap HASH(blue)=00100100 Different flows have different hash values

48. Bitmap counting – direct bitmap HASH(green)=10001001 Packets from the same flow always hash to the same bit

49. Bitmap counting – direct bitmap HASH(violet)=10010101 Collisions OK, estimates compensate for them

50. Bitmap counting – direct bitmap HASH(orange)=11110011

51. HASH(pink)=11100000

52. HASH(yellow)=01100011 As the bitmap fills up, estimates get inaccurate

53. Bitmap counting – direct bitmap Solution: use more bits HASH(green)=10001001

54. Bitmap counting – direct bitmap Solution: use more bits Problem: memory scales with the number of flows HASH(blue)=00100100

55. Bitmap counting – virtual bitmap Solution: a) store only a portion of the bitmap b) multiply estimate by scaling factor

56. Bitmap counting – virtual bitmap HASH(pink)=11100000

57. HASH(yellow)=01100011 Problem: estimate inaccurate when few flows active

58. Bitmap counting – multiple bmps Solution: use many bitmaps, each accurate for a different range

59. Bitmap counting – multiple bmps HASH(pink)=11100000

60. HASH(yellow)=01100011

61. Use this bitmap to estimate number of flows

62. Bitmap counting – multiple bmps Use this bitmap to estimate number of flows

63. Bitmap counting – multires. bmp Problem: must update up to three bitmaps per packet Solution: combine bitmaps into one OR

64. HASH(pink)=11100000 Bitmap counting – multires. bmp

65. HASH(yellow)=01100011

66. Multiresolution Bitmaps Still too expensive to scale Scaled bitmap  Recycles the hash space with too many bits set  Adjusts the scaling factor according E.g., 1 bit represents 2 flows as opposed to a single flow

67. Too CPU-Intensive A packet with 1,000 bytes of payload  Needs 960 fingerprints for string length of 40 Prone to Denial-of-Service attacks

68. CPU Scaling Obvious approach: sampling - Random sampling may miss many substrings Solution: value sampling  Track only certain substrings e.g. last 6 bits of fingerprint are 0  P(not tracking a worm) = P(not tracking any of its substrings)

69. CPU Scaling Example  Track only substrings with last 6 bits = 0s  String length = 40  1,000 char string 960 substrings  960 fingerprints ‘11100…101010’…‘10110…000000’…  Use only ‘xxxxx….000000’ as signatures Probably 960 / 2 6 = 15 signatures

70. CPU Scaling P(finding a 100-byte signature) = 55% P(finding a 200-byte signature) = 92% P(finding a 400-byte signature) = 99.64%

71. Putting It Together headerpayload substring fingerprints keysrc cntdest cnt AD entry exist? update counters keycnt else update counter cnt > prevalence threshold? create AD entry Content Prevalence Table Address Dispersion Table counters > dispersion threshold? report key as suspicious worm

72. Putting It Together Sample frequency: 1/64 String length: 40 Use 4 hash functions to update prevalence table  Multistage filter reset every 60 seconds

73. System Design Two major components  Sensors Sift through traffic for a given address space Report signatures  An aggregator Coordinates real-time updates Distributes signatures

74. Implementation and Environment Written in C and MySQL (5,000 lines) rrd-tools library for graphical reporting PHP scripting for administrative control Prototype executes on a 1.6Ghz AMD Opteron 242 1U Server  Linux 2.6 kernel

75. EarlyBird Processes 1TB of traffic per day Can keep up with 200Mbps of continuous traffic

76. Parameter Tuning Prevalence threshold: 3  Very few signatures repeat Address dispersion threshold  30 sources and 30 destinations  Reset every few hours  Reduces the number of reported signatures down to ~25,000

77. Parameter Tuning Tradeoff between and speed and accuracy  Can detect Slammer in 1 second as opposed to 5 seconds With 100x more reported signatures

78. Performance 200Mbps Can be pipelined and parallelized for achieve 40Gbps

79. Memory Consumption Prevalence table  4 stages Each with ~500,000 bins (8 bits/bin)  2MB total Address dispersion table  25K entries (28 bytes each)  < 1 MB Total: < 4MB

80. Trace-Based Verification Two main sources of false positives  2,000 common protocol headers e.g. HTTP, SMTP Whitelisted  SPAM e-mails  BitTorrent Many-to-many download

81. False Negatives So far none Detected every worm outbreak

82. Inter-Packet Signatures An attacker might evade detection by splitting an invariant string across packets With 7MB extra, EarlyBird can keep per flow states and fingerprint across packets

83. Live Experience with EarlyBird Detected precise signatures  CodeRed variants  MyDoom mail worm  Sasser  Kibvu.B

84. Variant Content Polymorphic viruses  Semantically equivalent but textually distinct code  Invariant decoding routine

85. Extensions Self configuration Slow worms

86. Containment How to handle false positives?  If too aggressive, EarlyBird becomes a target for DoS attacks  An attacker can fool the system to block a target message

87. Coordination Trust of deployed servers Validation Policy

88. Conclusions EarlyBird is a promising approach  To detect unknown worms real-time  To extract signatures automatically  To detect SPAMs with minor changes Wire-speed signature learning is viable