End-to-end Anomalous Event Detection in Production Networks Les Cottrell, Connie Logg, Felipe Haro, Mahesh Chhaparia (SLAC), Maxim Grigoriev (FNAL), Mark Sandford (Loughborough University) ESCC Meeting, Salt Lake City February, 2005 http://www.slac.stanford.edu/grp/scs/net/talk05/escc-feb05.ppt Network administrators and others need ways to be notified when there are significant, persistent anomalous changes in network performance that may require intervention. We have successfully implemented techniques (plateau algorithm and Kolmogorov-Smirnov) to find such step changes in time series measurements of network performance. However, if there are large seasonal changes (e.g. day/night, weekday/weekend) then the number of false positives or misses can be larger than desirable. We are thus working with FNAL to evaluate seasonal effect algorithms (e.g. Holt-Winters) on real production network performance measurements. We will present results on the performance of such algorithms as well as plans to evaluate methods to look at multiple metrics (e.g. capacity, available bandwidth, RTT, multiple paths) simultaneously using Principal Component Analysis. Partially funded by DOE/MICS for Internet End-to-end Performance Monitoring (IEPM)

Outline Why? Input data How? Results Conclusions & Futures First approaches The real world Results Conclusions & Futures

Uses of Techniques Automated problem identification: Alerts for network administrators, e.g. Bandwidth changes in time-series, iperf, SNMP Alerts for systems people OS/Host metrics Anomalies for security Forecasts (are a fallout of the techniques) for Grid Middleware, e.g. replica manager, data placement

Data Uses packet pair dispersion of 20 packets to provide: Capacity, X-traffic, available bandwidth At 3 minute intervals Very noisy time series data Moving averaged over 1 hour Capacity

Plateau, most intuitive Each observation: If outside history buffer mean mh ± b*sh then add to trigger buffer Else add to history, and remove oldest from trigger buffer When trigger buffer > t points then trigger issued Check if (mh - mt) / mh > D & 90% trigger in last T mins then have trigger Move trigger buffer to history buffer Observations Event * = history length = 1 day, t = trigger length = 3 hours = standard deviations = 2 We set the history buffer length to one day in order to minimize the lag between the history mean and the observations due to diurnal changes. Trigger % full History mean History mean – 2 * stdev

K-S (Kolmogorov-Smirnov) For each observation: for the previous 100 observations with next 100 observations Compare the vertical difference in CDFs How does it differ from random CDFs Expressed as % difference The trigger buffer reporting the event well after start of step down is partially an artifact. It could for example report the time of the start of the event as say when the trigger buffer reached 10% full. However, K-S is still more accurate in defining the time when the change was greatest. Compare K-S with Plateau

Compare Results between K-S & plateau very similar, using K-S threshold = 70% Current plateau only finds negative changes Useful to see when condition returns to normal K-S implemented in C and executes faster than Plateau (in Perl), depends on parameters K-S more formalized Plateau and K-S work well for non seasonal observations (e.g. small changes day/night) Plateau takes about 14 mins for 100 days H-W FNAL takes 7 mins K-S takes 1.08 min on +- 100 points 3.07 min on +- 200 points 14.75 min on +- 400 points

Seasons & false alerts Congestion on Monday following a quiet weekend causes a high forecast, gives an alert Also a history buffer of not a day causes History mean to be out of sync with observations

Diurnal Variation People arriving at work between 19:00 & 20:00 PDT (7:00 & 8:00 PK time) cause sudden drop in dynamic capacity

Effect on events Change in bandwidth (drops) between 19:00 & 20:00 causes more anomalous events around this time

Seasonal Changes Use Holt-Winters (H-W) technique: Uses triple exponential weighted moving average EWMA(i) = Obs(i) * a + (1-a) * EWMA(i-1) Three terms each with its own parameter (a, b, ) that take into account local smoothing, long term seasonal smoothing, and trends The trend component for our data is flat.

H-W Implementation Need regularly spaced data (else going back one season is difficult, and gets out of sync): Interpolate data: select bin size Average points in bin If no points in first week bin then get data from future weeks For following weeks, missing data bins filled from previous week Initial values for smoothing from NIST “Engineering Statistics Handbook” Choose parms by minimizing (1/N)Σ(Ft-yt)2 Ft=forecast for time t as function of parameters, yt = observation at time t A week is special and defines a cycle of seasons. We do nothing special with the day. Note we need a weeks worth of data to get going

H-W Implementation Three implementations evaluated (two new) FNAL (Maxim Grigoriev) Inspiration for evaluating this method Part of RRD (Brutlag) Limited control over what it produces and how it works SLAC Implemented NIST formulation, different formulation/parameter values from Brutlag/FNAL, also added minimize sums of squares to get parms

Events Can look at residuals (Ft – yt), or Χ2 Could use K-S or plateau on: residuals, or on the local smoothing (i.e. after removing long term seasonal effects)

Example Local smoothing 99% weight for last 24 hours Linear trend 50% last 24 hours Seasonal 99% for last week Within an 80 minute window, 80% points outside deviation envelope ≡ event Observations Deviations are smoothed absolute residuals Note the difference in weekend vs weekday Deviations Forecast Weekend Weekdays

Evaluation Created a library of time series for 100 days from June through Sep 2004 for 40 hosts Analyzed using Plateau and saved all events where trigger buffer filled (no filters on size of step) 23 hosts had 120 candidate events Event types: steps; diurnal changes; congestion from cron jobs, bandwidth tests, flash crowds Classify ~120 events as to whether interesting Large, sharp drop in bandwidth, persist for >> 3hrs Plateau easiest to understand and tune etc. also first to be developed. Classification is subjective, large (mh-mt)/mh> 10%, also looked at 30%, step occurs in <4 hours

Results K-S shows similar results to Plateau As adjust parameters to reduce false positives then increase missed events E.g. for plateau with trigger buffer = 3 hrs filled to 90% in < 220 minutes, history buffer=1 day, effect of threshold D=(mh-mt)/mh Plateau (b=2) K-S with ± 100 observations D False Miss 10% 16% 8% 30% 2% 32%

Conclusions A few paths (10%) have strong seasonal effects Plateau & K-S work well if only weak seasonal effects K-S detects both step downs & up, also gives accurate time estimate of event (good for correlations) H-W promising for seasonal effects, but Is more complex, and requires more parameters which may not be easy to estimate Requires regular data (interpolation step) CPU time can depend critically on parameters chosen, e.g. increasing K-S range from ±100 to say ±400 increases CPU time by factor 14 H-W works, still need to quantify its effectiveness Looking at PCA to evaluate multiple metrics simultaneously (e.g. fwd & bwd traffic, RTT, multiple paths)

More information SLAC Plateau implementation www.acm.org/sigs/sigcomm/sigcomm2004/workshop_papers/nts26-logg1.pdf SLAC H-W implementation www-iepm.slac.stanford.edu/monitoring/forecast/hw.html Eng. Statistics Handbook http://www.itl.nist.gov/div898/handbook/pmc/section4/pmc435.htm