Northwestern Lab for Internet and Security Technology (LIST) Yan Chen Router-based Anomaly/Intrusion Detection and Mitigation (RAIDM) Systems Scalable.

Slides:



Advertisements
Similar presentations
Sketch-based Change Detection Balachander Krishnamurthy (AT&T) Subhabrata Sen (AT&T) Yin Zhang (AT&T) Yan Chen (UCB/AT&T) ACM Internet Measurement Conference.
Advertisements

Code-Red : a case study on the spread and victims of an Internet worm David Moore, Colleen Shannon, Jeffery Brown Jonghyun Kim.
Hash-Based IP Traceback Best Student Paper ACM SIGCOMM’01.
Building Your Own Firewall Chapter 10. Learning Objectives List and define the two categories of firewalls Explain why desktop firewalls are used Explain.
Cyber Threat Analysis  Intrusions are actions that attempt to bypass security mechanisms of computer systems  Intrusions are caused by:  Attackers accessing.
1 Yan Chen Northwestern Lab for Internet and Security Technology (LIST) Dept. of Computer Science Northwestern University
1 Yan Chen Northwestern University Lab for Internet and Security Technology (LIST) in Northwestern.
1 Reversible Sketches for Efficient and Accurate Change Detection over Network Data Streams Robert Schweller Ashish Gupta Elliot Parsons Yan Chen Computer.
An Algebraic Approach to Practical and Scalable Overlay Network Monitoring Yan Chen, David Bindel, Hanhee Song, Randy H. Katz Presented by Mahesh Balakrishnan.
EECS Presentation Web Tap: Intelligent Intrusion Detection Kevin Borders.
Intrusion Detection/Prevention Systems. Objectives and Deliverable Understand the concept of IDS/IPS and the two major categorizations: by features/models,
Towards Scalable and Robust Distributed Intrusion Alert Fusion with Good Load Balancing Zhichun Li, Yan Chen and Aaron Beach Lab for Internet & Security.
Reverse Hashing for High-speed Network Monitoring: Algorithms, Evaluation, and Applications Robert Schweller 1, Zhichun Li 1, Yan Chen 1, Yan Gao 1, Ashish.
Welcome to EECS 354 Network Penetration and Security.
Yan Chen, Hai Zhou Northwestern Lab for Internet and Security Technology (LIST) Dept. of Electrical Engineering and Computer Science Northwestern University.
Reverse Hashing for Sketch Based Change Detection in High Speed Networks Ashish Gupta Elliot Parsons with Robert Schweller, Theory Group Advisor: Yan Chen.
© 2006 Cisco Systems, Inc. All rights reserved. Implementing Secure Converged Wide Area Networks (ISCW) Module 6: Cisco IOS Threat Defense Features.
UNCLASSIFIED Secure Indirect Routing and An Autonomous Enterprise Intrusion Defense System Applied to Mobile ad hoc Networks J. Leland Langston, Raytheon.
Welcome to CS 450 Internet Security: A Measurement-based Approach.
Towards a High-speed Router-based Anomaly/Intrusion Detection System (HRAID) Zhichun Li, Yan Gao, Yan Chen Northwestern.
A DoS Resilient Flow-level Intrusion Detection Approach for High-speed Networks Yan Gao, Zhichun Li, Yan Chen Lab for Internet and Security Technology.
1 Towards Anomaly/Intrusion Detection and Mitigation on High-Speed Networks Yan Gao, Zhichun Li, Manan Sanghi, Yan Chen, Ming- Yang Kao Northwestern Lab.
1 Network Intrusion Detection and Mitigation Yan Chen Northwestern Lab for Internet and Security Technology (LIST) Department of Computer Science Northwestern.
Intrusion Detection/Prevention Systems. Definitions Intrusion –A set of actions aimed to compromise the security goals, namely Integrity, confidentiality,
1 Towards Anomaly/Intrusion Detection and Mitigation on High-Speed Networks Yan Gao, Zhichun Li, Yan Chen Northwestern Lab for Internet and Security Technology.
1 Yan Chen Northwestern Lab for Internet and Security Technology (LIST) Dept. of Computer Science Northwestern University
Towards a High speed Router based Anomaly/Intrusion detection System Yan Gao & Zhichun Li.
Hash, Don’t Cache: Fast Packet Forwarding for Enterprise Edge Routers Minlan Yu Princeton University Joint work with Jennifer.
1 Network-based Intrusion Detection, Mitigation and Forensics System Yan Chen Department of Electrical Engineering and Computer Science Northwestern University.
Network-based and Attack-resilient Length Signature Generation for Zero-day Polymorphic Worms Zhichun Li 1, Lanjia Wang 2, Yan Chen 1 and Judy Fu 3 1 Lab.
1 HPNAIDM: the High-Performance Network Anomaly/Intrusion Detection and Mitigation System Yan Chen Lab for Internet & Security Technology (LIST) Department.
Denial of Service A Brief Overview. Denial of Service Significance of DoS in Internet Security Low-Rate DoS Attacks – Timing and detection – Defense High-Rate,
SCAN: a Scalable, Adaptive, Secure and Network-aware Content Distribution Network Yan Chen CS Department Northwestern University.
Final Introduction ---- Web Security, DDoS, others
Chapter 1. Introduction. By Sanghyun Ahn, Deot. Of Computer Science and Statistics, University of Seoul A Brief Networking History §Internet – started.
Scalable and Efficient Data Streaming Algorithms for Detecting Common Content in Internet Traffic Minho Sung Networking & Telecommunications Group College.
Global Intrusion Detection Using Distribute Hash Table Jason Skicewicz, Laurence Berland, Yan Chen Northwestern University 6/2004.
INTERNATIONAL NETWORKS At Indiana University Hans Addleman TransPAC Engineer, International Networks University Information Technology Services Indiana.
CSCI 530 Lab Intrusion Detection Systems IDS. A collection of techniques and methodologies used to monitor suspicious activities both at the network and.
The UCSD Network Telescope A Real-time Monitoring System for Tracking Internet Attacks Stefan Savage David Moore, Geoff Voelker, and Colleen Shannon Department.
Welcome to Introduction to Computer Security. Why Computer Security The past decade has seen an explosion in the concern for the security of information.
Mapping Internet Sensors with Probe Response Attacks Authors: John Bethencourt, Jason Franklin, Mary Vernon Published At: Usenix Security Symposium, 2005.
1 Network-based Intrusion Detection, Prevention and Forensics System Yan Chen Department of Electrical Engineering and Computer Science Northwestern University.
A Dos Resilient Flow-level Intrusion Detection Approach for High-speed Networks Yan Gao, Zhichun Li, Yan Chen Department of EECS, Northwestern University.
CINBAD CERN/HP ProCurve Joint Project on Networking 26 May 2009 Ryszard Erazm Jurga - CERN Milosz Marian Hulboj - CERN.
Securing the Network Infrastructure. Firewalls Typically used to filter packets Designed to prevent malicious packets from entering the network or its.
Anomaly/Intrusion Detection and Prevention in Challenging Network Environments 1 Yan Chen Department of Electrical Engineering and Computer Science Northwestern.
1 SOS: Secure Overlay Services A. D. Keromytis V. Misra D. Runbenstein Columbia University.
Secure In-Network Aggregation for Wireless Sensor Networks
1 Network Intrusion Detection and Mitigation Yan Chen Northwestern Lab for Internet and Security Technology (LIST) Department of Computer Science Northwestern.
Workpackage 3 New security algorithm design ICS-FORTH Ipswich 19 th December 2007.
Anomaly/Intrusion Detection and Prevention in Challenging Network Environments 1 Yan Chen Department of Electrical Engineering and Computer Science Northwestern.
Troubleshooting Mesh Networks Lili Qiu Joint Work with Victor Bahl, Ananth Rao, Lidong Zhou Microsoft Research Mesh Networking Summit 2004.
DoS/DDoS attack and defense
1 Modeling, Early Detection, and Mitigation of Internet Worm Attacks Cliff C. Zou Assistant professor School of Computer Science University of Central.
Yan Chen Dept. of Electrical Engineering and Computer Science Northwestern University Spring Review 2008 Award # : FA Intrusion Detection.
Network-based and Attack-resilient Length Signature Generation for Zero-day Polymorphic Worms Zhichun Li 1, Lanjia Wang 2, Yan Chen 1 and Judy Fu 3 1 Lab.
Monitoring, Diagnosing, and Securing the Internet 1 Yan Chen Department of Electrical Engineering and Computer Science Northwestern University Lab for.
Northwestern Lab for Internet & Security Technology (LIST)
Firewalls. Overview of Firewalls As the name implies, a firewall acts to provide secured access between two networks A firewall may be implemented as.
1 Scalability and Accuracy in a Large-Scale Network Emulator Nov. 12, 2003 Byung-Gon Chun.
Yan Chen Northwestern Lab for Internet and Security Technology (LIST) Dept. of Computer Science Northwestern University
Network Processing Systems Design
Network-based Intrusion Detection, Prevention and Forensics System
CHAPTER 3 Architectures for Distributed Systems
Northwestern Lab for Internet and Security Technology (LIST) Yan Chen Department of Computer Science Northwestern University.
Network Intrusion Detection and Mitigation
Yan Chen Lab for Internet & Security Technology (LIST)
End-user Based Network Measurement and Diagnosis
Presentation transcript:

Northwestern Lab for Internet and Security Technology (LIST) Yan Chen Router-based Anomaly/Intrusion Detection and Mitigation (RAIDM) Systems Scalable and Accurate Overlay Network Monitoring and Diagnosis Wireless and Ad hoc Networking

Northwestern Lab for Internet and Security Technology (LIST) Yan Chen Department of Computer Science Northwestern University

Internet is becoming a new infrastructure for service delivery –World wide web, –VoIP – –Interactive TV? Major challenges for Internet-scale services –Scalability: 600M users, 35M Web sites, 2.1Tb/s –Security: viruses, worms, Trojan horses, etc. –Mobility: ubiquitous devices in phones, shoes, etc. –Agility: dynamic systems/network, congestions/failures –Ossification: extremely hard to deploy new technology in the core Our Theme

Projects at LIST Global Router-based Anomaly/Intrusion Detection (GRAID) Systems Distributed Information Retrieval Systems

Battling Hackers is a Growth Industry! The past decade has seen an explosion in the concern for the security of information Internet attacks are increasing in frequency, severity and sophistication Denial of service (DoS) attacks –Cost $1.2 billion in 2000 –Thousands of attacks per week in 2001 –Yahoo, Amazon, eBay, Microsoft, White House, etc., attacked --Wall Street Journal (11/10/2004)

Battling Hackers is a Growth Industry (cont’d) Virus and worms faster and powerful –Melissa, Nimda, Code Red, Code Red II, Slammer … –Cause over $28 billion in economic losses in 2003, growing to over $75 billion in economic losses by –Code Red (2001): 13 hours infected >360K machines - $2.4 billion loss –Slammer (2003): 10 minutes infected > 75K machines - $1 billion loss Spywares are ubiquitous –80% of Internet computers have spywares installed

The Spread of Sapphire/Slammer Worms

Current Intrusion Detection Systems (IDS) Mostly host-based and not scalable to high-speed networks –Slammer worm infected 75,000 machines in <10 mins –Host-based schemes inefficient and user dependent »Have to install IDS on all user machines ! Mostly signature-based –Cannot recognize unknown anomalies/intrusions –New viruses/worms, polymorphism Statistical detection –Hard to adapt to traffic pattern changes –Unscalable for flow-level detection »IDS vulnerable to DoS attacks –Overall traffic based: inaccurate, high false positives

Current Intrusion Detection Systems (II) Cannot differentiate malicious events with unintentional anomalies –Anomalies can be caused by network element faults –E.g., router misconfiguration, signal interference of wireless network, etc. Isolated or centralized systems –Insufficient info for causes, patterns and prevalence of global-scale attacks

Global Router-based Anomaly/Intrusion Detection (GRAID) Systems Online traffic recording and analysis for high- speed networks –Leverage sketches for data streaming computation Online adaptive flow-level anomaly/intrusion detection and mitigation –Leverage statistical learning theory (SLT) adaptively learn the traffic pattern changes –E.g., busy vs. idle wireless networks, with different level of interferences, etc. –Unsupervised learning without knowing ground truth

GRAID Systems (II) Integrated approach for false positive reduction –Signature-based detection –Network element fault diagnostics –Traffic signature matching of emerging applications Hardware speedup for real-time detection –Collaborated with Gokhan Memik (ECE of NU) –Try various hardware platforms: FPGAs, network processors Scalable anomaly/intrusion alarm fusion with distributed hash tables (DHT) –Automatically distribute alerts with similar symptoms to the same fusion center for analysis

GRAID Detection Sensor Attached to a router or access point as a black box Edge network detection is particularly powerful Router LA N Inter net Switch LA N (a) Router LAN Inter net LA N (b) GRAID sensor scan port Splitter Router LA N Inter net LA N (c) Splitter GRAID sensor Switch GRAID sensor GRAID sensor Original configuration Monitor each port separately Monitor aggregated traffic from all ports

GRAID Sensor Architecture Reversible k-ary sketch monitoring Filtering Sketch based statistical anomaly detection (SSAD) Local sketch records Sent out for aggregation Remote aggregated sketch records Per-flow monitoring Streaming packet data Normal flows Suspicious flows Intrusion or anomaly alarms to fusion centers Keys of suspicious flows Keys of normal flows Data path Control path Modules on the critical path Signature -based detection Traffic profile checking Statistical detection Part I Sketch- based monitoring & detection Part II Per-flow monitoring & detection Modules on the non-critical path Network fault detection

Scalable Traffic Monitoring and Analysis - Challenge Potentially tens of millions of time series ! –Need to work at very low aggregation level (e.g., IP level) »Changes may be buried inside aggregated traffic –The Moore’s Law on traffic growth …  Per-flow analysis is too slow or too expensive –Want to work in near real time Existing approaches not directly applicable –Mostly focus on heavy-hitters

Sketch-based Change Detection (ACM SIGCOMM IMC 2003, 2004) Input stream: (key, update) Sketch module Forecast module(s) Change detection module (k,u) … Sketches Error Sketch Alarms Report flows with large forecast errors Summarize input stream using sketches Build forecast models on top of sketches

Sketch Probabilistic summary of data streams –Originated in STOC 1996 [AMS96] –Widely used in database research to handle massive data streams SpaceAccuracy Hash tablePer-key state100% SketchCompact With probabilistic guarantees (better for larger values)

K-ary Sketch Array of hash tables: T j [K] (j = 1, …, H) 1 j H 01K-1 … … … hj(k)hj(k) hH(k)hH(k) h1(k)h1(k) Update (k, u): T j [ h j (k)] += u (for all j)

K-ary Sketch (cont’d) Estimate v(S, k): sum of updates for key k compensate for signal loss v(S, k) + noise v(S, k)/K + E(noise) boost confidence unbiased estimator of v(S,k) with low variance 1 j H 01K-1 … … … hj(k)hj(k) hH(k)hH(k) h1(k)h1(k)

Forecast Model: EWMA Sketches are linear (Can combine sketches) Compute forecast error sketch: S error =    S forecast (t) S observed (t-1) S forecast (t-1) =  - S error (t-1) S observed (t-1) S forecast (t-1) Update forecast sketch: S forecast

Evaluated with tier-1 ISP trace and NU traces Scalable –Can handle tens of millions of time series Accurate –Provable probabilistic accuracy guarantees –Even more accurate on real Internet traces Efficient –For the worst case traffic, all 40 byte packets: »16 Gbps on a single FPGA board »526 Mbps on a Pentium-IV 2.4GHz PC –Only less than 3MB memory used Patent filed Evaluation of Reversible K-ary Sketch

Remaining Challenges Reversible sketch to infer the culprit flows (ACM SIGCOMM IMC 2004) Hierarchical and multi-dimensional sketch Detecting distributed and insidious attacks with sketch

GRAID Sensor Architecture Reversible k-ary sketch monitoring Filtering Sketch based statistical anomaly detection (SSAD) Local sketch records Sent out for aggregation Remote aggregated sketch records Per-flow monitoring Streaming packet data Normal flows Suspicious flows Intrusion or anomaly alarms to fusion centers Keys of suspicious flows Keys of normal flows Data path Control path Modules on the critical path Signature -based detection Traffic profile checking Statistical detection Part I Sketch- based monitoring & detection Part II Per-flow monitoring & detection Modules on the non-critical path Network fault detection

Statistical Anomaly Detection Online statistical detection with sketches Applying Statistical Learning Theory (STL) –Use Hidden Markov Model (HMM) to adaptively learn the parameters Focus on two major intrusions: denial of service (DoS) attacks and port scanning Monitor traffic with multiple sketches –With different keys »(Source IP, Dest IP) »(Source IP, Dest port) »(Dest IP, Dest port) –For each key, record the number of unconnected TCP requests: SYN – SYN/ACK

Intrusion Mitigation Attacks detectedMitigation Denial of Service (DoS), e.g., TCP SYN flooding SYN defender, SYN proxy, or SYN cookie for victim Port Scan and wormsIngress filtering with attacker IP Vertical port scanQuarantine the victim machine Horizontal port scanMonitor traffic with the same port # for compromised machine SpywaresWarn the end users being spied

GRAID Sensor Architecture Reversible k-ary sketch monitoring Filtering Sketch based statistical anomaly detection (SSAD) Local sketch records Sent out for aggregation Remote aggregated sketch records Per-flow monitoring Streaming packet data Normal flows Suspicious flows Intrusion or anomaly alarms to fusion centers Keys of suspicious flows Keys of normal flows Data path Control path Modules on the critical path Signature -based detection Traffic profile checking Statistical detection Part I Sketch- based monitoring & detection Part II Per-flow monitoring & detection Modules on the non-critical path Network fault detection

Network Diagnosis and Fault Location Infrastructure ossification led to thrust of overlay applications Traceroute gives hop-by-hop round-trip latency –Asymmetric routing – Can’t get hop-by-hop loss rate ! Network tomography –Infer the properties of links from end-to-end measurements –Limited measurements -> under-constrained system, unidentifiable links –Existing work uses various constraints and assumptions »Tree-like topology »The number of lossy links is small 12 1’ 1

Our Approach: Virtual Links Minimal link sequences (path segments) whose loss rates uniquely identified –Locate the faults to certain link(s) The first lower-bound on the network tomography granularity Use algebraic scheme to find virtual links –Leverage our work on overlay network monitoring (ACM SIGCOMM IMC 2003, ACM SIGCOMM 2004)

GRAID Sensor Architecture Reversible k-ary sketch monitoring Filtering Sketch based statistical anomaly detection (SSAD) Local sketch records Sent out for aggregation Remote aggregated sketch records Per-flow monitoring Streaming packet data Normal flows Suspicious flows Intrusion or anomaly alarms to fusion centers Keys of suspicious flows Keys of normal flows Data path Control path Modules on the critical path Signature -based detection Traffic profile checking Statistical detection Part I Sketch- based monitoring & detection Part II Per-flow monitoring & detection Modules on the non-critical path Network fault detection

Intrusion/anomaly Alarm Fusion Individual IDS has bad accuracy due to limited view Crucial to collect information from multiple vantage points – distributed IDS (DIDS) –Each IDS generate local symptom report, send to sensor fusion center (SFC) Help understand the prevalence, cause and patterns of global-scale attacks Existing DIDS –Centralized fusion –Distributed fusion with unscalable communication

GRAID Sensor Interconnection Though Cyber Disease DHT (distributed hash table) for alarm fusion –Scalability –Load balancing –Fault-tolerance – Intrusion correlation Internet IDS IDS + SFC GRAID Coverage Attack Injected Attack Injected CDDHT Mesh

Basic Operations of CDDHT put (disease_key, symptom report) –Send report to SFC attack_info = get (disease_key) –Query about certain attacks from SFC Each operation only O(n) hops –n is the total number of nodes in CDDHT

CDDHT: Disease Key Design IntrusionIDCharacterization Field(s) DoS Attack0Victim IP (subnet) Scans10 (for vertical & block scan) Source IP address Destination IP (for vertical scan) 0 (for block scan) 1 (for horizontal & coordinated scan) Scan port number Source IP (for horizontal scan) 0 (for coordinated scan) Viruses/Worms20 (for known virus/worm)Worm ID 1 (for unknown virus/worm)Destination port number

Other Challenges of CDDHT Load balancing Supporting complicated queries –E.g., aggregate queries Attack resilience –OK to have some IDS sensors compromised –What about SFCs?

Research methodology Combination of theory, synthetic/real trace driven simulation, and real-world implementation and deployment

Conclusion for GRAID Systems Online traffic recording and analysis on high- speed networks Online statistical anomaly detection Integrated approach for false positive reduction –Signature-based detection –Network element fault diagnostics –Traffic signature matching of emerging applications Hardware speedup for real-time detection Scalable anomaly/intrusion alarm fusion with distributed hash tables (DHT)