Analysis of Anomalous Payload-based Worm Detection and Signature Generation by Ke Wang, Gabriela Cretu, Salvatore J.Stolfo Columbia University.

Slides:



Advertisements
Similar presentations
Wenke Lee and Nick Feamster Georgia Tech Botnet and Spam Detection in High-Speed Networks.
Advertisements

Wenke Lee and Nick Feamster Georgia Tech Botnet and Spam Detection in High-Speed Networks.
Detecting Evasion Attacks at High Speeds without Reassembly Detecting Evasion Attacks at High Speeds without Reassembly George Varghese J. Andrew Fingerhut.
Greg Williams CS691 Summer Honeycomb  Introduction  Preceding Work  Important Points  Analysis  Future Work.
TransAD: A Content Based Anomaly Detector Sharath Hiremagalore Advisor: Dr. Angelos Stavrou October 23, 2013.
Worm Origin Identification Using Random Moonwalks Yinglian Xie, V. Sekar, D. A. Maltz, M. K. Reiter, Hui Zhang 2005 IEEE Symposium on Security and Privacy.
Polymorphic blending attacks Prahlad Fogla et al USENIX 2006 Presented By Himanshu Pagey.
 Looked at some research approaches to: o Evaluate defense effectiveness o Stop worm from spreading from a given host o Defend a circle of friends against.
 Well-publicized worms  Worm propagation curve  Scanning strategies (uniform, permutation, hitlist, subnet) 1.
Intrusion Detection Systems and Practices
Snort - an network intrusion prevention and detection system Student: Yue Jiang Professor: Dr. Bojan Cukic CS665 class presentation.
5/1/2006Sireesha/IDS1 Intrusion Detection Systems (A preliminary study) Sireesha Dasaraju CS526 - Advanced Internet Systems UCCS.
Usenix Security 2004 Autograph Toward Automated, Distributed Worm Signature Detection Hyang-Ah KimBrad Karp Carnegie Mellon UniversityIntel Research &
Worms: Taxonomy and Detection Mark Shaneck 2/6/2004.
© 2006 Cisco Systems, Inc. All rights reserved. Implementing Secure Converged Wide Area Networks (ISCW) Module 6: Cisco IOS Threat Defense Features.
Automated Worm Fingerprinting Sumeet Singh, Cristian Estan, George Varghese, and Stefan Savage Manan Sanghi.
FIREWALLS & NETWORK SECURITY with Intrusion Detection and VPNs, 2 nd ed. 6 Packet Filtering By Whitman, Mattord, & Austin© 2008 Course Technology.
Internet Quarantine: Requirements for Containing Self-Propagating Code David Moore et. al. University of California, San Diego.
Lecture 11 Intrusion Detection (cont)
FIREWALL TECHNOLOGIES Tahani al jehani. Firewall benefits  A firewall functions as a choke point – all traffic in and out must pass through this single.
Lucent Technologies – Proprietary Use pursuant to company instruction Learning Sequential Models for Detecting Anomalous Protocol Usage (work in progress)
Data Mining for Intrusion Detection: A Critical Review Klaus Julisch From: Applications of data Mining in Computer Security (Eds. D. Barabara and S. Jajodia)
1 Intrusion Detection Systems. 2 Intrusion Detection Intrusion is any use or attempted use of a system that exceeds authentication limits Intrusions are.
Penetration Testing Security Analysis and Advanced Tools: Snort.
A Statistical Anomaly Detection Technique based on Three Different Network Features Yuji Waizumi Tohoku Univ.
Differences between In- and Outbound Internet Backbone Traffic Wolfgang John and Sven Tafvelin Dept. of Computer Science and Engineering Chalmers University.
Fast Portscan Detection Using Sequential Hypothesis Testing Authors: Jaeyeon Jung, Vern Paxson, Arthur W. Berger, and Hari Balakrishnan Publication: IEEE.
Principles of Computer Security: CompTIA Security + ® and Beyond, Third Edition © 2012 Principles of Computer Security: CompTIA Security+ ® and Beyond,
Intrusion Detection and Prevention. Objectives ● Purpose of IDS's ● Function of IDS's in a secure network design ● Install and use an IDS ● Customize.
Distributed IDS The implementation of a Distributed Intrusion Detection System over a medium scale open network where the focus is availability of services.
CSCI 530 Lab Intrusion Detection Systems IDS. A collection of techniques and methodologies used to monitor suspicious activities both at the network and.
Click to add Text Automated Worm Fingerprinting Sumeet Singh, Cristian Estan, George Varghese and Stefan Savage Department of Computer Science and Engineering.
DoWitcher: Effective Worm Detection and Containment in the Internet Core S. Ranjan et. al in INFOCOM 2007 Presented by: Sailesh Kumar.
HONEYPOTS PRESENTATION TEAM: TEAM: Ankur Sharma Ashish Agrawal Elly Bornstein Santak Bhadra Srinivas Natarajan.
Vigilante: End-to-End Containment of Internet Worms Authors : M. Costa, J. Crowcroft, M. Castro, A. Rowstron, L. Zhou, L. Zhang, and P. Barham In Proceedings.
Modeling Worms: Two papers at Infocom 2003 Worms Programs that self propagate across the internet by exploiting the security flaws in widely used services.
Learning Rules for Anomaly Detection of Hostile Network Traffic Matthew V. Mahoney and Philip K. Chan Florida Institute of Technology.
Chapter 12 Transmission Control Protocol (TCP)
Automatically Generating Models for Botnet Detection Presenter: 葉倚任 Authors: Peter Wurzinger, Leyla Bilge, Thorsten Holz, Jan Goebel, Christopher Kruegel,
IEEE Communications Surveys & Tutorials 1st Quarter 2008.
Mapping Internet Sensors with Probe Response Attacks Authors: John Bethencourt, Jason Franklin, Mary Vernon Published At: Usenix Security Symposium, 2005.
Wide-scale Botnet Detection and Characterization Anestis Karasaridis, Brian Rexroad, David Hoeflin In First Workshop on Hot Topics in Understanding Botnets,
Finding Diversity in Remote Code Injection Exploits Justin Ma, John Dunagan, Helen J. Wang, Stefan Savage, Geoffrey M. Voelker *University of California,
DETECTING TARGETED ATTACKS USING SHADOW HONEYPOTS AUTHORS: K. G. Anagnostakisy, S. Sidiroglouz, P. Akritidis, K. Xinidis, E. Markatos, A. D. Keromytisz.
Polygraph: Automatically Generating Signatures for Polymorphic Worms James Newsome, Brad Karp, and Dawn Song Carnegie Mellon University Presented by Ryan.
Worm Defense Alexander Chang CS239 – Network Security 05/01/2006.
Search Worms, ACM Workshop on Recurring Malcode (WORM) 2006 N Provos, J McClain, K Wang Dhruv Sharma
Polygraph: Automatically Generating Signatures for Polymorphic Worms Presented by: Devendra Salvi Paper by : James Newsome, Brad Karp, Dawn Song.
Intrusion Detection Systems Paper written detailing importance of audit data in detecting misuse + user behavior 1984-SRI int’l develop method of.
Automated Worm Fingerprinting Authors: Sumeet Singh, Cristian Estan, George Varghese and Stefan Savage Publish: OSDI'04. Presenter: YanYan Wang.
I NTRUSION P REVENTION S YSTEM (IPS). O UTLINE Introduction Objectives IPS’s Detection methods Classifications IPS vs. IDS IPS vs. Firewall.
Mapping Internet Sensor With Probe Response Attacks Authors: John Bethencourt, Jason Franklin, and Mary Vernon. University of Wisconsin, Madison. Usenix.
Firewalls A brief introduction to firewalls. What does a Firewall do? Firewalls are essential tools in managing and controlling network traffic Firewalls.
HoneyStat: Local Worm Detection Using Honeypots David Dagon, Xinzhou Qin, Guofei Gu, Wenke Lee, et al from Georgia Institute of Technology Authors: The.
PANACEA: AUTOMATING ATTACK CLASSIFICATION FOR ANOMALY-BASED NETWORK INTRUSION DETECTION SYSTEMS Reporter : 鄭志欣 Advisor: Hsing-Kuo Pao.
Network-based and Attack-resilient Length Signature Generation for Zero-day Polymorphic Worms Zhichun Li 1, Lanjia Wang 2, Yan Chen 1 and Judy Fu 3 1 Lab.
Polygraph: Automatically Generating Signatures for Polymorphic Worms Authors: James Newsome (CMU), Brad Karp (Intel Research), Dawn Song (CMU) Presenter:
Usenix Security 2004 Autograph Toward Automated, Distributed Worm Signature Detection Hyang-Ah KimBrad Karp Carnegie Mellon UniversityIntel Research &
Spamming Botnets: Signatures and Characteristics Yinglian Xie, Fang Yu, Kannan Achan, Rina Panigrahy, Microsoft Research, Silicon Valley Geoff Hulten,
2009/6/221 BotMiner: Clustering Analysis of Network Traffic for Protocol- and Structure- Independent Botnet Detection Reporter : Fong-Ruei, Li Machine.
Network Anomaly Detection Using Autonomous System Flow Aggregates Thienne Johnson 1,2 and Loukas Lazos 1 1 Department of Electrical and Computer Engineering.
Internet Quarantine: Requirements for Containing Self-Propagating Code
POLYGRAPH: Automatically Generating Signatures for Polymorphic Worms
Authors – Johannes Krupp, Michael Backes, and Christian Rossow(2016)
Worm Origin Identification Using Random Moonwalks
Net 323 D: Networks Protocols
Data Mining & Machine Learning Lab
ITIS 6167/8167: Network and Information Security
Intrusion Detection Systems
Chapter 4: outline 4.1 Overview of Network layer data plane
Presentation transcript:

Analysis of Anomalous Payload-based Worm Detection and Signature Generation by Ke Wang, Gabriela Cretu, Salvatore J.Stolfo Columbia University

Topics ● Main Goals ● Payload based anamoly detection – PAYL Overview ● PAYL sensor system – phases ● Experiments and Results ● Related Work ● References ● Summary 10/29/06 Sireesha Dasaraju 2 PAYL

Main Goals ● Accurately detect ZERO-DAY worms. ● Automatically generate signatures that can be shared with other vulnerable systems. 10/29/06 Sireesha Dasaraju 3 PAYL

Payload-based anamoly detection PAYL - Overview ● Detect worms by analyzing the packet payload. – A model of “normal data” is maintained. – A new zero-day attack will have content data never seen by the victim host. – A newly infected host will begin sending outbound traffic that is very similar to the content it received. – Correlate ingress/egress anomalous payload alerts to detect the worm propagation. 10/29/06 Sireesha Dasaraju 4 PAYL

PAYL – Overview – continued ● Automatic signature generation. – Signatures generated based on correlated ingress/egresss content anamolies. – The overlapping content of the similar outgoing and incoming anomalous payloads determine the candidate worm signature. 10/29/06 Sireesha Dasaraju 5 PAYL

● Signature sharing – A central security system to be used by the coolaborating sites. – Any signature generated by any of the sites will be shared with the central system and will be exchanged with all the sites. – Each site can then update their onsite filtering rules. 10/29/06 Sireesha Dasaraju 6 PAYL PAYL – Overview – continued

PAYL sensor - Phases ● The PAYL sensor operates in the following phases – Modeling Normal Data – Calibration – Detection – Signature generation 10/29/06 Sireesha Dasaraju 7 PAYL

Modeling the normal content ● Assumption – packet content available for modeling. ● The technique used: – n-gram : A sequence of 'n' adjacent byte values in the packet payload. ( n = 1 for first implementation) – The frequency of each n-gram is computed. – This frequency represents the statistical centroid or model of the content flow. – The normalized average frequency and the variance of each gram are computed. – The byte value distribution is graphed. (Graph with the ASCII character on the x-axis and character frequency on the y-axis) 10/29/06 Sireesha Dasaraju 8 PAYL

Graphs ● 10/29/06 Sireesha Dasaraju 9 PAYL

Modeling the normal content - continued – A rank ordered distribution is then graphed. (Graph with the frequency count on x-axis and average character frequency on the y-axis) – A Z-string is determined from the rank ordered distribution. – A Z-string is a string of distinct bytes whose frequency in the data is ordered from frequent to least, ignoring those byte values that do not appear in the data. – The Z-String representation provides a privacy-preserving summary of the payload that may be exchanged between domains without revealing the true content. – Z-String mainly used for message exchange and cross domain correlation of alerts. 10/29/06 Sireesha Dasaraju 10 PAYL

Calibrating the sensor ● Calibration – A sample of test data is measured against the centroids and an initial value for a threshold setting is chosen. – Subsequent round of testing of new data updates the threshold settings to clibrate the sensor to the operating environment. – This way for each centroid, there is a distinct threshold value. 10/29/06 Sireesha Dasaraju 11 PAYL

Detection ● Detection – To compare the similarity between the actual data and the trained models, Mahalanobis distance technique is used. – In this technique, the mean frequency of the n-gram of the actual payload packet, is weighed against the centroid, to derive the difference in terms of a distance. – The distance is then compared to a threshold value. – If the distance greater than the threshold, an alert is issued. 10/29/06 Sireesha Dasaraju 12 PAYL

Signature Generation ● Technique for generating signatures : – When some incoming anomalous traffic to port i is detected, an ingress alert is generated and places the packet content on a buffer list of “suspects”. – Any outbound traffic from the port i is then compared to the buffer. – The comparision is done on the packet contents and a similarity score is computed. – If the score is higher than the threshold, this is a possible worm propagation and is blocked. 10/29/06 Sireesha Dasaraju 13 PAYL

Signature Generation - contd ● Packet comparsion Techniques : – String Equality ● Egress payload is exactly the same as the ingress suspect packet contents. ● Very strict, few false positives. ● But if the worm changes even a single bit or its packet fragmentation between the input and output ports, it cannot be detected. ● Similarity score is either 0 or 1. (1 -- equality) – Longest common substring (LCS) ● The longer the common substring the greater the confidence. ● Avoids the above fragmentation problem. 10/29/06 Sireesha Dasaraju 14 PAYL

Signature Generation - contd ● Computation overhead. ● String lengths L1 and L2; Common substring length C, the similarity score is 2 * C/(L1+L2) – Longest common subsequence ● The longest subsequence may not be contiguous ● Can detect the polymorphic worms, but too many false positives. ● String lengths L1 and L2; Common substring length C, the similarity score is 2 * C/(L1+L2) ● Each of the above techniques result in some similarity score and will be compared against the threshold. ● The common substring found will serve as the worm. 10/29/06 Sireesha Dasaraju 15 PAYL

Experiments and Results ● Data Used – Three distinct real world datasets. – Worm Set - CodeRed, CodeRedII, WebDav and a worm that exploits the IIS windows media service. ● Data preparation – Each dataset is split into two distinct portions, one for training and the other for testing. – For each test dataset, a clean set of packets, free of any known worms, is created. – Into this clean test data, a set of worm data is inserted at the random places. 10/29/06 Sireesha Dasaraju 16 PAYL

Results ● 10/29/06 Sireesha Dasaraju 17 PAYL

Results ● PAYL detected all the worms at a very low false positive rate. – For 0.1% false positive rate, ● First Data Set resulted in 5.8 alerts per hour. ● Second Data set resulted in 6 alerts per hour. ● Third Data set resulted in 8 alerts per hour. ● Tested the detection rate of W32.Blaster worm on TCP 135 port, using real RPC traffic inside Columbia's CS department. – The worm packets were detected with zero false positives. 10/29/06 Sireesha Dasaraju 18 PAYL

Related Work ● Rule-based network intrusion detection (eg. Snort) – Depend on the signatures. – Signatures can be generated only after the worm has been launched successfully. – The time between the worm launch and its wide-spread infestation is very short and is not enough time to generate the signatures for filtering and to patch the vulnerable systems. – Will miss the brand new attacks. ● Sensors based on scan and probe activity – Detects based on network packet header analysis or monitoring the connection attempts and traffic volume. ● Will miss the slow-propagating worms. ● Will miss the attacks carrying malicious content in an otherwise normal connection. 10/29/06 Sireesha Dasaraju 19 PAYL

● Shield – Detection based on vulnerability signatures instead of the string-oriented content signatures. – Vulnerability signatures specify what an exploit would look like in the datagram of packets – A host based shield agent would drop any connections that match this specification. – Time tag to specify, test and deploy shields. 10/29/06 Sireesha Dasaraju 20 PAYL

Related Work - continued ● Honeycomb – Host-based intrusion detection system. – Automatically generate the signatures. – Uses honeypot to capture malicious traffic targetting dark space. – Applies the longest common substring algorithm on the packet content of a number of connections going to the same services. – The computed substring is a candidate worm signature. 10/29/06 Sireesha Dasaraju 21 PAYL

Related work - continued ● Autograph – Classifies traffic into two categories, a flow pool with suspicious scanning activity and a non-suspicious flow pool – TCP flow reassembly is applied to the suspicious flow pool and apply Rabin fingerprints to partition the payload into small blocks. – The most frequent substrings from these blocks form a worm signature. – Blacklisting is used in order to decrease the number of false positives. – Suspicious IPs and destination ports are exchanged between the multiple sensors at the collaborating sites. 10/29/06 Sireesha Dasaraju 22 PAYL

Related Work - continued ● Earlybird – Similar to Autograph system. – The substrings computed by Rabin fingerprints are Are maintained in a frequency count table, incrementing a count field each time the substring is encountered. – The information about source and destination Ips are recorded. – The table is sorted by the order of frequency counts. – To keep the false positives down, IP address dispersion is applied by counting the distinct source and destination IPs for each suspicious content. 10/29/06 Sireesha Dasaraju 23 PAYL

Summary ● PAYL can detect worms without signatures, so can detect the Zero-day worms. ● Correlating the content of the ingress and egress alerts will reduce the false positives. ● PAYL can generate detailed content signatures. ● PAYL combined with centralized security system can help all the collaborating sites stay up-to-date on the latest worm signatures. ● PAYL handles the zero-day worms better than the other detection systems mentioned in the related work. 10/29/06 Sireesha Dasaraju 24 PAYL

References ● K Wang, Gabriela Cretu, Salvatore J.Stoflo, Anomalous payload-based network intrusion detection, in Proceedings of Recent Advance in Intrusion Detection (RAID), Sept ● C.Kreibich and J.Crowcroft. Honeycomb-Creating Intrusion Detection Signatures Using Honeypots, In Proceedings of the 2 nd Workshop on Hot Topics in Networks (HotNets-II), November 2003 ● M.Locasto, J.Parekh, S.Stolfo, A.Keromytis, T.Malkin and V.Misra. Collaborative Distributed Intrusion Detection, Columbia University Tech Report CUCS ,2004 ● H.J.Wang, C.Guo, D.R.Simon, and A.Zugenmaier. Shield: Vulnerability-Driven Network Filter for Preventing Known Vulnerability Exploits. In Proceedings of the ACM SIGCOMM Conference, Aug.2004 ● K-A Kim and B.Karp. Autograph: toward Automated Distributed Worm distribution, In Proceedings of the USENIX Security Symposium, August ● S.Singh, C.Estan, G.Varghese and S.Savage. Automated Worm Fingerprinting, Sixth Symposium on Operating Systems Design and Implementation (OSDI), /29/06 Sireesha Dasaraju 25 PAYL