Distributed Network Monitoring in the Wisconsin Advanced Internet Lab Paul Barford Computer Science Department University of Wisconsin – Madison Spring, 2002

Motivation Many applications that run over the Internet have minimum performance requirements The network is one of the two possible sources of poor performance Wide area network behavior is unpredictable –IP networks are best effort –Constant change is normal Quality of service capability is not widely deployed –Will it ever be available?

Monitoring is a First Step Accurate monitoring of network state can enable application adaptivity and improved network management –Data provides basis for improved models and protocols There are many challenges in network monitoring –All features of the Internet make monitoring difficult –When, where, what, how… Today’s focus 1.Network monitoring efforts at Wisconsin 2.Combining monitoring and analysis to understand network traffic anomalies

The Wisconsin Advanced Internet Lab Next generation environment for network research –Our focus: performance, management, security –Platform for testbeds: storage, grid computing, … Internal environment –Instances of end-to-end-through-core Internet paths External environment –Measurement nodes deployed across the Internet

WAIL’s External Environment Existing infrastructure –WAWM systems (10) –Surveyor systems (60) Partnership with Advanced Systems –NIMI systems (45) Partnership with PCS and ICIR –Condor/Grid Infrastructures Prototype system is under development Passive flow measurements –FlowScan data from UW, Internet2, others(?)

WAIL’s Internal Environment Complement to external facilities Hands-on test bed which creates paths identical to those in the Internet from end-to-end-through-core –Variety of highly configurable equipment Why do we need an internal lab? –Enables instrumentation and measurement of entire end-to-end system –Enables new systems and protocols to be implemented in places where access is not possible in wide area Vision of internal lab: New means for doing network research Status: Significant commitment from industry partners (Cisco, EMC, Fujitsu) and the university – rev. 1.0 by 5/1/02

Distributed Anomaly Detection Motivation: Anomaly detection and identification is an important task for network operators –Operators typically monitor by eye using SNMP or IP flows –Simple thresholding is ineffective –Some anomalies are obvious, other are not Focus: Characterize and develop distributed means for detecting classes of anomalies –Network outages, Flash crowds, Attacks, Measurement failures Approach: Use statistical and wavelet techniques to analyze anomalies from IP flow and SNMP data from UW and other sites Implications: Tools and infrastructure which quickly and accurately identify and adapt to traffic anomalies

Characteristics of “Normal” traffic

Our Approach to Analysis Analyze examples of each type of anomaly via statistics, time series and wavelets (our initial focus) Wavelets provide a means for describing time series data that considers both frequency and scale –Particularly useful for characterizing data with sharp spikes and discontinuities More robust than Fourier analysis which only shows what frequencies exist in a signal –Tricky to determine which wavelets provide best resolution of signals in data We use tools developed at UW Wavelet IDR center First step: Identify which filters isolate anomalies

Analysis of “Normal” Traffic Wavelets easily localize familiar daily/weekly signals

Example Anomaly: Attacks DoS: sharp increase in flows and/or packets in one direction Linear splines seem to be a good filter to distinguish DoS attacks

Characteristics of Flash Crowds Sharp increase in packets/bytes/flows followed by slow return to normal behavior eg. Linux releases Leading edge not significantly different from DoS signal so next step is to look within the spikes

Characteristics of Network Anomalies Typically a steep drop off in packets/bytes/flows followed a short time later by restoration

Summary and Conclusion Accurate network monitoring is essential for improving application performance and network management The Wisconsin Advanced Internet Lab provides a unique environment for network monitoring Wavelets are an effective means for identifying anomalous behavior in data gathered from IP flow and SNMP interface monitors –Details on distributed and coordinated monitoring and analysis available this spring