Advanced Broadband Communications Center (CCABA) Universitat Politècnica de Catalunya (UPC) SMARTxAC: A Passive Monitoring and Analysis System for High-Speed.

Slides:



Advertisements
Similar presentations
Abstract There is significant need to improve existing techniques for clustering multivariate network traffic flow record and quickly infer underlying.
Advertisements

Monitoring very high speed links Gianluca Iannaccone Sprint ATL joint work with: Christophe Diot – Sprint ATL Ian Graham – University of Waikato Nick McKeown.
Live migration of Virtual Machines Nour Stefan, SCPD.
SHARKFEST '09 | Stanford University | June 15–18, 2009 The Reality of 10G Analysis Presented by: Network Critical Wednesday, June 17 th, :30 pm –
New Directions in Traffic Measurement and Accounting Cristian Estan – UCSD George Varghese - UCSD Reviewed by Michela Becchi Discussion Leaders Andrew.
LONG: Laboratories Over Next Generation Networks. LONG Laboratories Over Next Generation Networks Jordi Domingo-Pascual Josep Mangues-Bafalluy Advanced.
Centre de Comunicacions Avançades de Banda Ampla (CCABA) Universitat Politècnica de Catalunya (UPC) Identification of Network Applications based on Machine.
A Data Stream Management System for Network Traffic Management Shivnath Babu Stanford University Lakshminarayanan Subramanian Univ. California, Berkeley.
A Flexible Model for Resource Management in Virtual Private Networks Presenter: Huang, Rigao Kang, Yuefang.
Merit Network: Connecting People and Organizations Since 1966 CALEA Compliance – A Feasibility Study October 25, 2006 Mary Eileen McLaughlin Director –
Nick Duffield, Patrick Haffner, Balachander Krishnamurthy, Haakon Ringberg Rule-Based Anomaly Detection on IP Flows.
ARP Traffic Study Jim Rees, Manish Karir Research and Development Merit Network Inc.
Performance Analysis of Orb Rabin Karki and Thangam V. Seenivasan 1.
Monitoring a Large-Scale Network: Selecting the Right Tool Sayadur Rahman United International University & Network Manager, Financial Service.
1 Virtual Machine Resource Monitoring and Networking of Virtual Machines Ananth I. Sundararaj Department of Computer Science Northwestern University July.
5/1/2006Sireesha/IDS1 Intrusion Detection Systems (A preliminary study) Sireesha Dasaraju CS526 - Advanced Internet Systems UCCS.
CHINA EDUCATION & RESEARCH NETWORK CENTER Linuxflow: A High Speed Backbone Measurement Facility ZhiChun Li Hui Zhang.
Towards a High-speed Router-based Anomaly/Intrusion Detection System (HRAID) Zhichun Li, Yan Gao, Yan Chen Northwestern.
NetFlow Analyzer Drilldown to the root-QoS Product Overview.
A Signal Analysis of Network Traffic Anomalies Paul Barford, Jeffrey Kline, David Plonka, and Amos Ron.
Licentiate Seminar: On Measurement and Analysis of Internet Backbone Traffic Wolfgang John Department of Computer Science and Engineering Chalmers University.
Cross Strait Quad-Regional Radio Science and Wireless Technology Conference, Vol. 2, p.p. 980 – 984, July 2011 Cross Strait Quad-Regional Radio Science.
Lecture 11 Intrusion Detection (cont)
Sven Ubik, CESNET TNC2004, Rhodos, 9 June 2004 Performance monitoring of high-speed networks from NREN perspective.
Sven Ubik, Petr Žejdl CESNET TNC2008, Brugges, 19 May 2008 Passive monitoring of 10 Gb/s lines with PC hardware.
Reading Report 14 Yin Chen 14 Apr 2004 Reference: Internet Service Performance: Data Analysis and Visualization, Cross-Industry Working Team, July, 2000.
WARNINGBIRD: A Near Real-time Detection System for Suspicious URLs in Twitter Stream.
Traffic Classification through Simple Statistical Fingerprinting M. Crotti, M. Dusi, F. Gringoli, L. Salgarelli ACM SIGCOMM Computer Communication Review,
Net Optics Confidential and Proprietary Net Optics appTap Intelligent Access and Monitoring Architecture Solutions.
Cross-Domain Privacy-Preserving Cooperative Firewall Optimization.
NICE :Network Intrusion Detection and Countermeasure Selection in Virtual Network Systems.
Dynamic Resource Allocation Using Virtual Machines for Cloud Computing Environment.
SIGCOMM 2002 New Directions in Traffic Measurement and Accounting Focusing on the Elephants, Ignoring the Mice Cristian Estan and George Varghese University.
Test Of Distributed Data Quality Monitoring Of CMS Tracker Dataset H->ZZ->2e2mu with PileUp - 10,000 events ( ~ 50,000 hits for events) The monitoring.
POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (1) 2. Network Monitoring Metrics.
POSTECH DP&NM Lab. Internet Traffic Monitoring and Analysis: Methods and Applications (1) 5. Passive Monitoring Techniques.
NetFlow: Digging Flows Out of the Traffic Evandro de Souza ESnet ESnet Site Coordinating Committee Meeting Columbus/OH – July/2004.
High Performance Computing Processors Felix Noble Mirayma V. Rodriguez Agnes Velez Electric and Computer Engineer Department August 25, 2004.
Vladimír Smotlacha CESNET Full Packet Monitoring Sensors: Hardware and Software Challenges.
On the processing time for detection of Skype traffic P.M. Santiago del Río, J. Ramos, J.L. García-Dorado, J. Aracil Universidad Autónoma de Madrid A.
Abstract Link error and malicious packet dropping are two sources for packet losses in multi-hop wireless ad hoc network. In this paper, while observing.
MASCOTS 2003 An Active Traffic Splitter Architecture for Intrusion Detection Ioannis Charitakis Institute of Computer Science Foundation of Research And.
TAAD - A Tool for Traffic Analysis and Automatic Diagnosis Kathy L. Benninger NLANR/Pittsburgh Supercomputing Center.
Metadata Management of Terabyte Datasets from an IP Backbone Network: Experience and Challenges Sue B. Moon and Timothy Roscoe.
TASHKENT UNIVERSITY OF INFORMATION TECHNOLOGIES Lesson №18 Telecommunication software design for analyzing and control packets on the networks by using.
Bandwidth Distributed Denial of Service: Attacks and Defenses.
4/19/20021 TCPSplitter: A Reconfigurable Hardware Based TCP Flow Monitor David V. Schuehler.
Vladimír Smotlacha CESNET High-speed Programmable Monitoring Adapter.
Preventing Private Information Inference Attacks on Social Networks.
Efficient Cache Structures of IP Routers to Provide Policy-Based Services Graduate School of Engineering Osaka City University
U N I V E R S I T Y O F S O U T H F L O R I D A Hadoop Alternative The Hadoop Alternative Larry Moore 1, Zach Fadika 2, Dr. Madhusudhan Govindaraju 2 1.
Centre de Comunicacions Avançades de Banda Ampla (CCABA) Universitat Politècnica de Catalunya (UPC) Identification of Network Applications based on Machine.
“OpenCALEA” Pragmatic Cost Effective CALEA Compliance Manish Karir, Merit - Research and Development.
3G wireless system  Speeds from 125kbps-2Mbps  Performance in computer networking (WCDMA, WLAN Bluetooth) & mobile devices area (cell.
#16 Application Measurement Presentation by Bobin John.
Sven Ubik, Aleš Friedl CESNET TNC 2009, Malaga, Spain, 11 June 2009 Experience with passive monitoring deployment in GEANT2 network.
© 2014 Level 3 Communications, LLC. All Rights Reserved. Proprietary and Confidential. Simple, End-to-End Performance Management Application Performance.
Online School Management System Supervisor Name: Ashraful Islam Juwel Lecturer of Asian University of Bangladesh Submitted By: Bikash Chandra SutrodhorID.
Connect communicate collaborate Performance Metrics & Basic Tools Robert Stoy, DFN EGI TF, Madrid September 2013.
1 Netflow Collection and Aggregation in the AT&T Common Backbone Carsten Lund.
Distributed Network Monitoring in the Wisconsin Advanced Internet Lab Paul Barford Computer Science Department University of Wisconsin – Madison Spring,
NetFlow Analyzer Best Practices, Tips, Tricks. Agenda Professional vs Enterprise Edition System Requirements Storage Settings Performance Tuning Configure.
“OpenCALEA” Pragmatic Cost Effective CALEA Compliance
Advanced Troubleshooting with Cisco Prime NAM-3: Use Case
Distributed Network Traffic Feature Extraction for a Real-time IDS
Securing the Network Perimeter with ISA 2004
Northwestern Lab for Internet and Security Technology (LIST) Yan Chen Department of Computer Science Northwestern University.
DDoS Attack Detection under SDN Context
IP Control Gateway (IPCG)
Presentation transcript:

Advanced Broadband Communications Center (CCABA) Universitat Politècnica de Catalunya (UPC) SMARTxAC: A Passive Monitoring and Analysis System for High-Speed Networks TERENA Networking Conference 2006 Pere Barlet-Ros Josep Solé-Pareta Javier Barrantes Eva Codina Jordi Domingo-Pascual {pbarlet, pareta, jbarranp, ecodina, Acknowledgment: This work has been partially supported by CESCA (SMARTxAC agreement) and the Spanish MEC (ref. TSI C03-02)

SMARTxAC SMARTxAC: Traffic Monitoring and Analysis System for the Anella Científica  Operative since July 2003  Developed under a collaboration agreement CESCA-UPC  Tailor-made traffic monitoring system for the Anella Científica Main objectives  Low-cost platform  Continuous monitoring of high-speed links without packet loss  Detection of network anomalies and irregular usage  Multi-user system: Network operators and Institutions Measurement of two full-duplex GigE links  Connection between Anella Científica and RedIRIS  Current load: ≈ 1.5 Gbps / ≈ 270 Kpps

Anella Científica Measurement point 2 x GigE full-duplex

Daily Network Usage

System Architecture Monitoring high-speed links is challenging  Collection of Gbps and storage of Terabytes of data per day  Limitations of current technology –CPU power, memory access speeds, bus and disk bandwidth, storage capacity, etc. Tailor-made system divided according to real-time constraints and running on different computers  Capture System (severe real-time constraints)  Traffic Analysis System (soft real-time constraints)  Result Visualization System (user driven) Data reduction: Early discard unnecessary information  Improve performance  Reduce storage requirements

Measurement Scenario dag0 dag1 REDIRIS Other Regional Nodes ESPANIX GÉANT Capture System (DAG 4.3GE + GPS) Traffic Analysis System (Linux) Result Visualization System Private network 2 Gbps CISCO 6513 (Anella Científica) Juniper M-20 (RedIRIS) RedIRIS (Madrid) Internet Connection 2 x 2Gbps ANELLA CIENTÍFICA RedIRIS Global Internet Management network

Capture System Capture hardware  Intel Xeon 2.4 GHz. + 1 GB. RAM  2 x Endace DAG 4.3GE  4 x Optical splitters  Precise timestamping using GPS (Trimble Acutime 2000) Capture software  Multi-threaded implementation  Collection of packet-headers without loss (no sampling)  5-tuple flow aggregation  Aggregated flows are sent to the Analysis System Data Reduction  Header collection: ≈1:10(90 GB/min  9 GB/min)  Flow aggregation: ≈1:200(45 GB/5 min  200 MB/5 min)  Some data is kept to analyze anomalies (window of ≈ 20 GB.)

Measurement Scenario dag0 dag1 REDIRIS Other Regional Nodes ESPANIX GÉANT Capture System (DAG 4.3GE + GPS) Traffic Analysis System Result Visualization System Private network 2 Gbps CISCO 6513 (Anella Científica) Juniper M-20 (RedIRIS) RedIRIS (Madrid) Internet Connection 2 x 2Gbps ANELLA CIENTÍFICA RedIRIS Global Internet Management network

Traffic Analysis System Analysis hardware  Pentium IV 2.6 GHz. + 1 GB. RAM Analysis Software  Aggregation of 5-tuple flows into classified flows –  –Origins: Institutions (also Network access points) –Destinations: External networks RedIRIS is connected to –Bidirectional aggregation  This classification can be useful for charging/cost-sharing Data reduction  Classified flows: >1:1000 (≈ 60 GB/day  ≈ 50 MB/day)  Compared with header traces: > 1: (≈ 13 TB/day)

Measurement Scenario dag0 dag1 REDIRIS Other Regional Nodes ESPANIX GÉANT Capture System (DAG 4.3GE + GPS) Traffic Analysis System Result Visualization System Private network 2 Gbps CISCO 6513 (Anella Científica) Juniper M-20 (RedIRIS) RedIRIS (Madrid) Internet Connection 2 x 2Gbps ANELLA CIENTÍFICA RedIRIS Global Internet Management network

Result Visualization System Hardware  Pentium III 450 MHz. Software  Web-based graphical interface  Institutions only have access to their own statistics  Graphs are generated on demand Available graphs  More than 300 combinations of graphs per institution and day  Statistics are updated every 5 minutes  Also weekly, monthly and yearly reports

Use case 1: Port Scanning Traffic profile per application (bps)

Use case 1: Port Scanning Traffic profile per application (flows/s)

Use case 1: Port Scanning Destination port: MySQL (tcp/3306) SRC IPDST IPSRC PORTDST PORT A.B C.D A.B.45.75E.F A.B C.D A.B C.D A.B C.D A.B C.D A.B C.D A.B.45.75E.F A.B C.D A.B.45.75E.F A.B.45.75E.F A.B C.D A.B.45.75E.F A.B C.D A.B.45.75E.F

Use case 2: Warez Server Traffic profile per application (bps)

Use case 2: Warez Server Top-10 (bytes)

Use case 3: Denial-of-Service Traffic profile per application (bps)

Anomaly Detection Threshold-based anomaly detection  An upper and lower traffic threshold can be set per institution  Thresholds: bits/sec, packets/sec and flows/sec  Different intervals: day/night and workday/weekend  Once an anomaly is detected additional information is kept –Additional information can be reviewed later offline Profile-based anomaly detection (work in progress)  Time-series prediction (adaptive linear filter)  It is not needed to know the “ordinary” traffic profile  Anomalies are detected when actual traffic differs from its predicted value  Thresholds mitigate limitations of adaptive prediction with long- term anomalies

Identification of Network Applications Traffic classification in SMARTxAC is based on port numbers  Port-based classification is no longer reliable  P2P, dynamic ports, tunnelling, web-based services, … We are developing a classification method based on machine learning techniques  It learns features of traffic flows that identify a given application  Packet payloads are only needed in the training phase  Once the system is trained only packet headers are needed

Preliminary Results (Accuracy)

Port-based vs. Machine Learning Port-based Machine learning

Conclusions SMARTxAC is a tailor-made network monitoring system that  Operates at gigabit speeds without packet loss  It is relatively low-cost  Provides very detailed information about the network usage  Multi-user system: network operators and institutions Since 2003, SMARTxAC is daily used by CESCA to detect anomalies, attacks, performance problems, network faults, etc. Future work  Anomaly detection and application identification  Sampling, IPv6 support, …  Deployment of more measurement points in the Anella Científica  Release the source code under an open-source license  Collaboration with Intel’s CoMo:

Advanced Broadband Communications Center (CCABA) Universitat Politècnica de Catalunya (UPC) SMARTxAC: A Passive Monitoring and Analysis System for High-Speed Networks TERENA Networking Conference 2006 Pere Barlet-Ros Josep Solé-Pareta Javier Barrantes Eva Codina Jordi Domingo-Pascual {pbarlet, pareta, jbarranp, ecodina, Acknowledgment: This work has been partially supported by CESCA (SMARTxAC agreement) and the Spanish MEC (ref. TSI C03-02)