Analysis of Anomalous Payload-based Worm Detection and Signature Generation by Ke Wang, Gabriela Cretu, Salvatore J.Stolfo Columbia University

Topics ● Main Goals ● Payload based anamoly detection – PAYL Overview ● PAYL sensor system – phases ● Experiments and Results ● Related Work ● References ● Summary 10/29/06 Sireesha Dasaraju 2 PAYL

Main Goals ● Accurately detect ZERO-DAY worms. ● Automatically generate signatures that can be shared with other vulnerable systems. 10/29/06 Sireesha Dasaraju 3 PAYL

Payload-based anamoly detection PAYL - Overview ● Detect worms by analyzing the packet payload. – A model of “normal data” is maintained. – A new zero-day attack will have content data never seen by the victim host. – A newly infected host will begin sending outbound traffic that is very similar to the content it received. – Correlate ingress/egress anomalous payload alerts to detect the worm propagation. 10/29/06 Sireesha Dasaraju 4 PAYL

PAYL – Overview – continued ● Automatic signature generation. – Signatures generated based on correlated ingress/egresss content anamolies. – The overlapping content of the similar outgoing and incoming anomalous payloads determine the candidate worm signature. 10/29/06 Sireesha Dasaraju 5 PAYL

● Signature sharing – A central security system to be used by the coolaborating sites. – Any signature generated by any of the sites will be shared with the central system and will be exchanged with all the sites. – Each site can then update their onsite filtering rules. 10/29/06 Sireesha Dasaraju 6 PAYL PAYL – Overview – continued

PAYL sensor - Phases ● The PAYL sensor operates in the following phases – Modeling Normal Data – Calibration – Detection – Signature generation 10/29/06 Sireesha Dasaraju 7 PAYL

Modeling the normal content ● Assumption – packet content available for modeling. ● The technique used: – n-gram : A sequence of 'n' adjacent byte values in the packet payload. ( n = 1 for first implementation) – The frequency of each n-gram is computed. – This frequency represents the statistical centroid or model of the content flow. – The normalized average frequency and the variance of each gram are computed. – The byte value distribution is graphed. (Graph with the ASCII character on the x-axis and character frequency on the y-axis) 10/29/06 Sireesha Dasaraju 8 PAYL

Graphs ● 10/29/06 Sireesha Dasaraju 9 PAYL

Modeling the normal content - continued – A rank ordered distribution is then graphed. (Graph with the frequency count on x-axis and average character frequency on the y-axis) – A Z-string is determined from the rank ordered distribution. – A Z-string is a string of distinct bytes whose frequency in the data is ordered from frequent to least, ignoring those byte values that do not appear in the data. – The Z-String representation provides a privacy-preserving summary of the payload that may be exchanged between domains without revealing the true content. – Z-String mainly used for message exchange and cross domain correlation of alerts. 10/29/06 Sireesha Dasaraju 10 PAYL

Calibrating the sensor ● Calibration – A sample of test data is measured against the centroids and an initial value for a threshold setting is chosen. – Subsequent round of testing of new data updates the threshold settings to clibrate the sensor to the operating environment. – This way for each centroid, there is a distinct threshold value. 10/29/06 Sireesha Dasaraju 11 PAYL

Detection ● Detection – To compare the similarity between the actual data and the trained models, Mahalanobis distance technique is used. – In this technique, the mean frequency of the n-gram of the actual payload packet, is weighed against the centroid, to derive the difference in terms of a distance. – The distance is then compared to a threshold value. – If the distance greater than the threshold, an alert is issued. 10/29/06 Sireesha Dasaraju 12 PAYL

Signature Generation ● Technique for generating signatures : – When some incoming anomalous traffic to port i is detected, an ingress alert is generated and places the packet content on a buffer list of “suspects”. – Any outbound traffic from the port i is then compared to the buffer. – The comparision is done on the packet contents and a similarity score is computed. – If the score is higher than the threshold, this is a possible worm propagation and is blocked. 10/29/06 Sireesha Dasaraju 13 PAYL

Signature Generation - contd ● Packet comparsion Techniques : – String Equality ● Egress payload is exactly the same as the ingress suspect packet contents. ● Very strict, few false positives. ● But if the worm changes even a single bit or its packet fragmentation between the input and output ports, it cannot be detected. ● Similarity score is either 0 or 1. (1 -- equality) – Longest common substring (LCS) ● The longer the common substring the greater the confidence. ● Avoids the above fragmentation problem. 10/29/06 Sireesha Dasaraju 14 PAYL

Signature Generation - contd ● Computation overhead. ● String lengths L1 and L2; Common substring length C, the similarity score is 2 * C/(L1+L2) – Longest common subsequence ● The longest subsequence may not be contiguous ● Can detect the polymorphic worms, but too many false positives. ● String lengths L1 and L2; Common substring length C, the similarity score is 2 * C/(L1+L2) ● Each of the above techniques result in some similarity score and will be compared against the threshold. ● The common substring found will serve as the worm. 10/29/06 Sireesha Dasaraju 15 PAYL

Experiments and Results ● Data Used – Three distinct real world datasets. – Worm Set - CodeRed, CodeRedII, WebDav and a worm that exploits the IIS windows media service. ● Data preparation – Each dataset is split into two distinct portions, one for training and the other for testing. – For each test dataset, a clean set of packets, free of any known worms, is created. – Into this clean test data, a set of worm data is inserted at the random places. 10/29/06 Sireesha Dasaraju 16 PAYL

Results ● 10/29/06 Sireesha Dasaraju 17 PAYL

Results ● PAYL detected all the worms at a very low false positive rate. – For 0.1% false positive rate, ● First Data Set resulted in 5.8 alerts per hour. ● Second Data set resulted in 6 alerts per hour. ● Third Data set resulted in 8 alerts per hour. ● Tested the detection rate of W32.Blaster worm on TCP 135 port, using real RPC traffic inside Columbia's CS department. – The worm packets were detected with zero false positives. 10/29/06 Sireesha Dasaraju 18 PAYL

Related Work ● Rule-based network intrusion detection (eg. Snort) – Depend on the signatures. – Signatures can be generated only after the worm has been launched successfully. – The time between the worm launch and its wide-spread infestation is very short and is not enough time to generate the signatures for filtering and to patch the vulnerable systems. – Will miss the brand new attacks. ● Sensors based on scan and probe activity – Detects based on network packet header analysis or monitoring the connection attempts and traffic volume. ● Will miss the slow-propagating worms. ● Will miss the attacks carrying malicious content in an otherwise normal connection. 10/29/06 Sireesha Dasaraju 19 PAYL

● Shield – Detection based on vulnerability signatures instead of the string-oriented content signatures. – Vulnerability signatures specify what an exploit would look like in the datagram of packets – A host based shield agent would drop any connections that match this specification. – Time tag to specify, test and deploy shields. 10/29/06 Sireesha Dasaraju 20 PAYL

Related Work - continued ● Honeycomb – Host-based intrusion detection system. – Automatically generate the signatures. – Uses honeypot to capture malicious traffic targetting dark space. – Applies the longest common substring algorithm on the packet content of a number of connections going to the same services. – The computed substring is a candidate worm signature. 10/29/06 Sireesha Dasaraju 21 PAYL

Related work - continued ● Autograph – Classifies traffic into two categories, a flow pool with suspicious scanning activity and a non-suspicious flow pool – TCP flow reassembly is applied to the suspicious flow pool and apply Rabin fingerprints to partition the payload into small blocks. – The most frequent substrings from these blocks form a worm signature. – Blacklisting is used in order to decrease the number of false positives. – Suspicious IPs and destination ports are exchanged between the multiple sensors at the collaborating sites. 10/29/06 Sireesha Dasaraju 22 PAYL

Related Work - continued ● Earlybird – Similar to Autograph system. – The substrings computed by Rabin fingerprints are Are maintained in a frequency count table, incrementing a count field each time the substring is encountered. – The information about source and destination Ips are recorded. – The table is sorted by the order of frequency counts. – To keep the false positives down, IP address dispersion is applied by counting the distinct source and destination IPs for each suspicious content. 10/29/06 Sireesha Dasaraju 23 PAYL

Summary ● PAYL can detect worms without signatures, so can detect the Zero-day worms. ● Correlating the content of the ingress and egress alerts will reduce the false positives. ● PAYL can generate detailed content signatures. ● PAYL combined with centralized security system can help all the collaborating sites stay up-to-date on the latest worm signatures. ● PAYL handles the zero-day worms better than the other detection systems mentioned in the related work. 10/29/06 Sireesha Dasaraju 24 PAYL

References ● K Wang, Gabriela Cretu, Salvatore J.Stoflo, Anomalous payload-based network intrusion detection, in Proceedings of Recent Advance in Intrusion Detection (RAID), Sept ● C.Kreibich and J.Crowcroft. Honeycomb-Creating Intrusion Detection Signatures Using Honeypots, In Proceedings of the 2 nd Workshop on Hot Topics in Networks (HotNets-II), November 2003 ● M.Locasto, J.Parekh, S.Stolfo, A.Keromytis, T.Malkin and V.Misra. Collaborative Distributed Intrusion Detection, Columbia University Tech Report CUCS ,2004 ● H.J.Wang, C.Guo, D.R.Simon, and A.Zugenmaier. Shield: Vulnerability-Driven Network Filter for Preventing Known Vulnerability Exploits. In Proceedings of the ACM SIGCOMM Conference, Aug.2004 ● K-A Kim and B.Karp. Autograph: toward Automated Distributed Worm distribution, In Proceedings of the USENIX Security Symposium, August ● S.Singh, C.Estan, G.Varghese and S.Savage. Automated Worm Fingerprinting, Sixth Symposium on Operating Systems Design and Implementation (OSDI), /29/06 Sireesha Dasaraju 25 PAYL