1 Towards Anomaly/Intrusion Detection and Mitigation on High-Speed Networks Yan Gao, Zhichun Li, Manan Sanghi, Yan Chen, Ming- Yang Kao Northwestern Lab.

Slides:



Advertisements
Similar presentations
Intrusion Detection Systems (I) CS 6262 Fall 02. Definitions Intrusion Intrusion A set of actions aimed to compromise the security goals, namely A set.
Advertisements

Sketch-based Change Detection Balachander Krishnamurthy (AT&T) Subhabrata Sen (AT&T) Yin Zhang (AT&T) Yan Chen (UCB/AT&T) ACM Internet Measurement Conference.
Code-Red : a case study on the spread and victims of an Internet worm David Moore, Colleen Shannon, Jeffery Brown Jonghyun Kim.
New Directions in Traffic Measurement and Accounting Cristian Estan – UCSD George Varghese - UCSD Reviewed by Michela Becchi Discussion Leaders Andrew.
Hamsa: Fast Signature Generation for Zero-day Polymorphic Worms with Provable Attack Resilience Zhichun Li, Manan Sanghi, Yan Chen, Ming-Yang Kao and Brian.
1 Yan Chen Northwestern Lab for Internet and Security Technology (LIST) Dept. of Computer Science Northwestern University
High-Performance Network Anomaly/Intrusion Detection & Mitigation System (HPNAIDM) Yan Chen Department of Electrical Engineering and Computer Science Northwestern.
1 Reversible Sketches for Efficient and Accurate Change Detection over Network Data Streams Robert Schweller Ashish Gupta Elliot Parsons Yan Chen Computer.
5/1/2006Sireesha/IDS1 Intrusion Detection Systems (A preliminary study) Sireesha Dasaraju CS526 - Advanced Internet Systems UCCS.
Intrusion Detection/Prevention Systems. Objectives and Deliverable Understand the concept of IDS/IPS and the two major categorizations: by features/models,
Northwestern Lab for Internet and Security Technology (LIST) Yan Chen Router-based Anomaly/Intrusion Detection and Mitigation (RAIDM) Systems Scalable.
Reverse Hashing for High-speed Network Monitoring: Algorithms, Evaluation, and Applications Robert Schweller 1, Zhichun Li 1, Yan Chen 1, Yan Gao 1, Ashish.
Yan Chen, Hai Zhou Northwestern Lab for Internet and Security Technology (LIST) Dept. of Electrical Engineering and Computer Science Northwestern University.
Statistical based IDS background introduction. Statistical IDS background Why do we do this project Attack introduction IDS architecture Data description.
High-Performance Network Anomaly/Intrusion Detection & Mitigation System (HPNAIDM) Zhichun Li Lab for Internet & Security Technology (LIST) Department.
Reverse Hashing for Sketch Based Change Detection in High Speed Networks Ashish Gupta Elliot Parsons with Robert Schweller, Theory Group Advisor: Yan Chen.
UNCLASSIFIED Secure Indirect Routing and An Autonomous Enterprise Intrusion Defense System Applied to Mobile ad hoc Networks J. Leland Langston, Raytheon.
Internet Cache Pollution Attacks and Countermeasures Yan Gao, Leiwen Deng, Aleksandar Kuzmanovic, and Yan Chen Electrical Engineering and Computer Science.
Towards a High-speed Router-based Anomaly/Intrusion Detection System (HRAID) Zhichun Li, Yan Gao, Yan Chen Northwestern.
Automated Worm Fingerprinting Sumeet Singh, Cristian Estan, George Varghese, and Stefan Savage Manan Sanghi.
High-Performance Network Anomaly/Intrusion Detection & Mitigation System (HPNAIDM) Yan Chen Lab for Internet & Security Technology (LIST) Department of.
A DoS Resilient Flow-level Intrusion Detection Approach for High-speed Networks Yan Gao, Zhichun Li, Yan Chen Lab for Internet and Security Technology.
1 Network Intrusion Detection and Mitigation Yan Chen Northwestern Lab for Internet and Security Technology (LIST) Department of Computer Science Northwestern.
What Learned Last Week Homework qn –What machine does the URL go to?
Intrusion Detection/Prevention Systems. Definitions Intrusion –A set of actions aimed to compromise the security goals, namely Integrity, confidentiality,
High-Performance Network Anomaly/Intrusion Detection & Mitigation System (HPNAIDM) Yan Chen Department of Electrical Engineering and Computer Science Northwestern.
1 Towards Anomaly/Intrusion Detection and Mitigation on High-Speed Networks Yan Gao, Zhichun Li, Yan Chen Northwestern Lab for Internet and Security Technology.
1 Yan Chen Northwestern Lab for Internet and Security Technology (LIST) Dept. of Computer Science Northwestern University
Towards a High speed Router based Anomaly/Intrusion detection System Yan Gao & Zhichun Li.
Testing Intrusion Detection Systems: A Critic for the 1998 and 1999 DARPA Intrusion Detection System Evaluations as Performed by Lincoln Laboratory By.
Lecture 11 Intrusion Detection (cont)
Department Of Computer Engineering
1 Network-based Intrusion Detection, Mitigation and Forensics System Yan Chen Department of Electrical Engineering and Computer Science Northwestern University.
Network-based and Attack-resilient Length Signature Generation for Zero-day Polymorphic Worms Zhichun Li 1, Lanjia Wang 2, Yan Chen 1 and Judy Fu 3 1 Lab.
Network-based and Attack-resilient Length Signature Generation for Zero-day Polymorphic Worms Zhichun Li 1, Lanjia Wang 2, Yan Chen 1 and Judy Fu 3 1 Lab.
1 HPNAIDM: the High-Performance Network Anomaly/Intrusion Detection and Mitigation System Yan Chen Lab for Internet & Security Technology (LIST) Department.
Intrusion Detection/Prevention Systems. Objectives and Deliverable Understand the concept of IDS/IPS and the two major categorizations: by features/models,
Computer Security: Principles and Practice First Edition by William Stallings and Lawrie Brown Lecture slides by Lawrie Brown Chapter 8 – Denial of Service.
Network Flow-Based Anomaly Detection of DDoS Attacks Vassilis Chatzigiannakis National Technical University of Athens, Greece TNC.
SCAN: a Scalable, Adaptive, Secure and Network-aware Content Distribution Network Yan Chen CS Department Northwestern University.
SIGCOMM 2002 New Directions in Traffic Measurement and Accounting Focusing on the Elephants, Ignoring the Mice Cristian Estan and George Varghese University.
Scalable and Efficient Data Streaming Algorithms for Detecting Common Content in Internet Traffic Minho Sung Networking & Telecommunications Group College.
Source-End Defense System against DDoS attacks Fu-Yuan Lee, Shiuhpyng Shieh, Jui-Ting Shieh and Sheng Hsuan Wang Distributed System and Network Security.
Hamsa: Fast Signature Generation for Zero-day Polymorphic Worms with Provable Attack Resilience Zhichun Li, Manan Sanghi, Yan Chen, Ming-Yang Kao and Brian.
Automatically Generating Models for Botnet Detection Presenter: 葉倚任 Authors: Peter Wurzinger, Leyla Bilge, Thorsten Holz, Jan Goebel, Christopher Kruegel,
IEEE Communications Surveys & Tutorials 1st Quarter 2008.
Mapping Internet Sensors with Probe Response Attacks Authors: John Bethencourt, Jason Franklin, Mary Vernon Published At: Usenix Security Symposium, 2005.
1 Network-based Intrusion Detection, Prevention and Forensics System Yan Chen Department of Electrical Engineering and Computer Science Northwestern University.
A Dos Resilient Flow-level Intrusion Detection Approach for High-speed Networks Yan Gao, Zhichun Li, Yan Chen Department of EECS, Northwestern University.
CINBAD CERN/HP ProCurve Joint Project on Networking 26 May 2009 Ryszard Erazm Jurga - CERN Milosz Marian Hulboj - CERN.
Anomaly/Intrusion Detection and Prevention in Challenging Network Environments 1 Yan Chen Department of Electrical Engineering and Computer Science Northwestern.
Chapter 7 Denial-of-Service Attacks Denial-of-Service (DoS) Attack The NIST Computer Security Incident Handling Guide defines a DoS attack as: “An action.
Department of Computer Science and Engineering Applied Research Laboratory Architecture for a Hardware Based, TCP/IP Content Scanning System David V. Schuehler.
1 Network Intrusion Detection and Mitigation Yan Chen Northwestern Lab for Internet and Security Technology (LIST) Department of Computer Science Northwestern.
Anomaly/Intrusion Detection and Prevention in Challenging Network Environments 1 Yan Chen Department of Electrical Engineering and Computer Science Northwestern.
Polygraph: Automatically Generating Signatures for Polymorphic Worms Presented by: Devendra Salvi Paper by : James Newsome, Brad Karp, Dawn Song.
Intrusion Detection Systems Paper written detailing importance of audit data in detecting misuse + user behavior 1984-SRI int’l develop method of.
Selective Packet Inspection to Detect DoS Flooding Using Software Defined Networking Author : Tommy Chin Jr., Xenia Mountrouidou, Xiangyang Li and Kaiqi.
1 Modeling, Early Detection, and Mitigation of Internet Worm Attacks Cliff C. Zou Assistant professor School of Computer Science University of Central.
Network-based and Attack-resilient Length Signature Generation for Zero-day Polymorphic Worms Zhichun Li 1, Lanjia Wang 2, Yan Chen 1 and Judy Fu 3 1 Lab.
Monitoring, Diagnosing, and Securing the Internet 1 Yan Chen Department of Electrical Engineering and Computer Science Northwestern University Lab for.
Northwestern Lab for Internet & Security Technology (LIST)
Yan Chen Northwestern Lab for Internet and Security Technology (LIST) Dept. of Computer Science Northwestern University
Network-based Intrusion Detection, Prevention and Forensics System
Northwestern Lab for Internet and Security Technology (LIST) Yan Chen Department of Computer Science Northwestern University.
Yan Chen Department of Electrical Engineering and Computer Science
Network Intrusion Detection and Mitigation
Yan Chen Lab for Internet & Security Technology (LIST)
Introduction to Internet Worm
Presentation transcript:

1 Towards Anomaly/Intrusion Detection and Mitigation on High-Speed Networks Yan Gao, Zhichun Li, Manan Sanghi, Yan Chen, Ming- Yang Kao Northwestern Lab for Internet and Security Technology (LIST) Department of Computer Science Northwestern University

2 Outline Motivation Architecture of RAIDM Statistical Sketch-based Anomaly/Intrusion Detection –Design –Evaluation Results Polymorphic Worm Signature Generation with Provable Attack Resilience –Design –Preliminary Evaluation Results Conclusions

3 Desired Features of Intrusion Detection Systems (IDS) Network-based and scalable to high-speed links –Slammer worm infected 75,000 machines in <10 mins –Flash worm can take less than 1 second to compromise 1M vulnerable machines in the Internet [Staniford04] –Host-based schemes inefficient and user dependent »Have to install IDS on all user machines ! –Existing network IDS unscalable: In a 10Gbps link, each 40-byte packet only has 32ns for processing ! Aggregated detection over multiple vantage points –Multi-homing, load balancing, policy routing become popular asymmetric routing –Even worse… Per-packet load balancing –Cannot afford to move traffic around

4 Features of Demanded Intrusion Detection Systems (IDS) Attack resiliency –Human adversaries are difficult to handle. Attacks tend to fool or DoS the IDS system. –Many IDSs use exact per-flow states, which is vulnerable to DoS attack. Attack root cause analysis for mitigation –Detection is only the first step. We also need mitigation and prevention. –Overall traffic based approaches can scale to high speed but cannot really help mitigation –We need flow-level information for mitigation –Signature generation for polymorphic worms

5 Outline Motivation Architecture of RAIDM Statistical Sketch-based Anomaly/Intrusion Detection –Design –Evaluation Results Polymorphic Worm Signature Generation with Provable Attack Resilience –Design –Preliminary Evaluation Results Conclusions

6 Router-based Anomaly/Intrusion Detection and Mitigation System (RAIDM) Online traffic recording –Design reversible sketch for data streaming computation –Record millions of flows (GB traffic) in a few hundred KB Online flow-level anomaly/intrusion detection & mitigation –As a first step, detect TCP SYN flooding, horizontal and vertical scans even when mixed »Existing schemes like TRW/AC, CPM will have high false positives –Infer key characteristics of malicious flows for mitigation Dos attack resiliency Apply to distributed detection environment RAIDM: First flow-level intrusion detection that can sustain 10s Gbps bandwidth even for worst case traffic of 40-byte packet streams

7 Router-based Anomaly/Intrusion Detection and Mitigation System (RAIDM) Attach to routers as a black box Router LA N Internet Switch (a) Router LA N (b) RAIDM system scan port Splitter Router (c) Splitter HAIDM system RAIDM system RAIDM system Switch LA N Internet Attach HRAID black boxes to high-speed routers (a) original configuration, (b) distributed configuration for which each port is monitored separately, (c) aggregate configuration for which a splitter is used to aggregate the traffic from all the ports of a router.

8 Reversible k-ary sketch monitoring Filtering Sketch based statistical anomaly detection (SSAD) Local sketch records Sent out for aggregation Remote aggregated sketch records Per-flow monitoring Streaming packet data Normal flows Suspicious flows Intrusion or anomaly alarms Keys of suspicious flows Keys of normal flows Data path Control path Modules on the critical path Signature -based detection Polymorphic worm detection (Hamsa) Part I Sketch- based monitoring & detection Part II Per-flow monitoring & detection Modules on the non-critical path Network fault diagnosis (DOD) RAIDM Architecture

9 Outline Motivation Architecture of RAIDM Statistical Sketch-based Anomaly/Intrusion Detection –Design –Evaluation Results Polymorphic Worm Signature Generation with Provable Attack Resilience –Design –Preliminary Evaluation Results Conclusions

10 Statistical Sketch-based Anomaly/Intrusion Detection (SSAD) Recording stage: record traffic of each router with different sketches, then transfer and combine them to current reversible sketch. Detection stage: use Time Series Analysis (Holt-Winter and EWMA) to detect large forecast error as anomalies. False positive reduction & 2D sketch skipped (lack of time)

11 Statistical Sketch-based Anomaly/Intrusion Detection (SSAD) Distributed Detection Environment

12 Background: Reversible k-ary Sketch Array of hash tables: Tj[K] (j = 1, …, H) –Similar to count sketch, counting bloom filter, multi- stage filter, … Update (k, u): T j [ h j (k)] += u (for all j) 1 j H 01K-1 … … … 1 j H 01 … … … hj(k)hj(k) hH(k)hH(k) h1(k)h1(k)

13 Background: Reversible k-ary Sketch + = Estimate v(S, k): sum of updates for key k Sketches are linear –Can Combine Sketches –Can aggregate data from different times, locations, and sources Inference I(S, t): output the heavy keys whose values are larger than threshold t

14 RS((DIP, Dport), SYN-SYN/ACK) RS((SIP, DIP), SYN-SYN/ACK) RS((SIP, Dport), SYN-SYN/ACK) Attack typesRS((DIP, Dport), SYN-SYN/ACK) RS((SIP, DIP), SYN-SYN/ACK) RS((SIP, Dport), SYN-SYN/ACK) SYN floodingYes Vertical scansNoYesNo Horizontal scansNo Yes Detection Algorithm

15 DoS Resiliency Analysis Possible DoS attack to existing approaches like TRW and TRW/AC –Source spoofed SYN flooding attack –Source spoofed packets to random location Attack SSAD is difficult. Possible attack is based on creating collisions in sketches. But… –Reverse engineering of hash functions is difficult –The possibility of finding collisions through exhaustive search is very low –Attacks are limited even with collisions: need the cooperation with comprised internal hosts.

16 Data Sets –NU traces (239M flows, 1.8TB traffic/day) –Lawrence Berkeley National Laboratory (LBL) Trace (900M flows) Rest of results based on NU trace evaluation Scalability and memory usage –Total 9.4MB used for recording hundreds of millions of flows »3 reversible k-ary sketches, 2 two dimensional sketches and 1 original k-ary sketch –Small # of memory access per packet Evaluation

17 Fast –Recording speed for the worst case traffic, all 40B pkts »16 Gbps on a single FPGA board »526 Mbps on a Pentium-IV 2.4GHz PC –Detection speed »On site NU experiment covering 1430 minutes: 0.34 sec for one minute on average. (std=0.64 sec) Accurate Anomaly Detection w/ Reversible Sketch –Compared with detection using complete flow-level logs –Provable probabilistic accuracy guarantees –Even more accurate on real Internet traces Evaluation (cont’d)

18 Evaluation (cont’d) 25 SYN flooding, 936 horizontal scans and 19 vertical scans detected (after sketch-based false positive reduction) 18 out of 25 SYN flooding verified w/ backscatter –Complete flow-level connection info used for backscatter Scans verified (all for vscan, top and bottom 10 for hscan) –Unknown scans also found in DShield and other alert reports Top 10 horizontal scans Bottom 10 horizontal scans DescriptionDportcount SQLSnake14335 W32.Rahack48992 unknown scan61011 Scan SSH221 MySQL Bot scans DescriptionDportcount Sasser or Korgo worm4453 W32.Sasser.B.Worm55541 Nachi or MSBlast worm1353 NetBIOS scan1393

19 Outline Motivation Architecture of RAIDM Statistical Sketch-based Anomaly/Intrusion Detection –Design –Evaluation Results Polymorphic Worm Signature Generation with Provable Attack Resilience –Design –Preliminary Evaluation Results Conclusions

20 Requirements for polymorphic worm signature generation Network-based –Worm spread at exponential speed, at early stage there are limited worm samples. –Keep up with the network speed ! Noise tolerant –Most network level flow classifier may suffer false positives. Attack resilience –Attackers always try to evade the signature generation system Efficient signature matching –Again, on high-speed networks! No existing work satisfies these requirements

21 Hamsa Content based v.s. behavior based signature –Content based: treat a worm as a byte sequence –Behavior based: the actual dynamics of the worm execution –Fast matching could be a problem for behavior based We propose Hamsa, a network-based signature generation system to meet the aforementioned requirements –Content based system (token based) –Accuracy comparable to the state-of-the art technique [Polygraph] but much faster –Propose an adversary model and has attack resilience guarantee

22 Hamsa Design Sniff traffic from networks Assembly the packets to flows Classify traffic based protocol Filter out known worms Generate suspicious and normal pools.

23 Hamsa Design Iterative signature generation Extract the tokens form suspicious pool Identify the same set of tokens in normal pool Core part: greedy algorithm based on the token information

24 Model-based Polymorphic Worm Signature Generation Problem & Algorithm No noise: O(n) Noise: NP-Hard The problem The model The algorithm

25 Evaluation Methodology Data collection –Normal pool: »300MB Web traffic for training »20GB Web traffic and Binary distribution of Linux for evaluation –Suspicious pool: »10 ~ 500 samples (with noise and different worms) »5000 samples each worm as false negative testing data set –Three pseudo worms: ATPhttpd, Apache-Knacker, Codered II –Two polymorphic engines from Internet Simulation settings –Single worm with noise »Test pool size 100 and 200 »Noise ratio 0, 10%, 30%, 50% –Multiple worms with noise »3 worms together: ATPhttpd, Apache-Knacker, Codered II »Test pool size 100 and 200 »Noise ratio 0, 10% and 25%

26 Preliminary Evaluation Results Signature Quality Signature Generation Speed –64 ~ 361 times faster than Polygraph –Only use several seconds to at most minutes to generate signatures Attack Resilience –Provable attack resilience (details omitted) –For token-fit attack, Polygraph fail but Hamsa succeed

27 Backup Slides

28 Intrusion Detection and Mitigation Attacks detectedMitigation Denial of Service (DoS), e.g., TCP SYN flooding SYN defender, SYN proxy, or SYN cookie for victim Port Scan and wormsIngress filtering with attacker IP Vertical port scanQuarantine the victim machine Horizontal port scanMonitor traffic with the same port # for compromised machine SpywaresWarn the end users being spied

29 Reversible Sketch Based Anomaly Detection Input stream: (key, update) (e.g., SIP, SYN-SYN/ACK) Sketch module Forecast module(s) Anomaly detection module (k,u) … Sketches Error Sketch Alarms Report flows with large forecast errors Infer the (characteristics) key for mitigation Summarize input stream using sketches Build forecast models on top of sketches

30 Reducing False Positives for SYN Flooding Detection Reasons of false positives of SYN flooding detection –Network/server congestions/failtures –Polluted or outdated DNS entries Filters to reduce false positives caused by bursty network/server congestions/failures Lifetime > Threshold life Filters to reduce the false positives caused by misconfigurations or related problems –No connection history

31 Intrusion Classification with Two Dimensional Sketch We need to distinguish different types of attack to take the most effective mitigation scheme –However, one dimensional information is not enough Non-spoofed SYN flooding v.s. horizontal scan {SIP,DP} –Bi-modal distribution. Two dimensional Sketch Structure of 2D sketch Example UPDATE

32 Intrusion Classification with Two Dimensional Sketch Two dimensional sketch is accurate –Refer to the paper about accuracy proof –Can archive % detection rate by the parameter mentioned in the paper

33 Automated worm signature generation Manual signature generation is too slow for fast propagated worms Automatic signature generation has been proposed [Earlybird][Autograph] But it is not hard for hackers to apply polymorphism to their worms. Polymorphic worm signature generation becomes necessary.