Self-Similar Traffic COMP5416 Advanced Network Technologies

Courtesy of Ashish Gupta (cs.northwestern.edu) 2 Why Self-Similarity? Trace data not consistent with queueing models

On the Self-Similar Nature of Ethernet Traffic Will E. Leland, Walter Willinger and Daniel V. Wilson Bellcore Murad S. Taqqu Boston University The Classic Paper:

Courtesy of Ashish Gupta (cs.northwestern.edu) 4 Overview What is Self Similarity? Ethernet Traffic is Self-Similar Source of Self Similarity Implications of Self Similarity

Courtesy of Ashish Gupta (cs.northwestern.edu) 5 Intuition of Self-Similarity Something “feels the same” regardless of scale

Courtesy of Ashish Gupta (cs.northwestern.edu) 6

7

8 Stochastic Objects In case of stochastic objects like time-series, self-similarity is used in the distributional sense  their mean, variance, correlation etc.

Courtesy of Ashish Gupta (cs.northwestern.edu) 9 Pictorial View of Self-Similarity

Courtesy of Ashish Gupta (cs.northwestern.edu) 10 Why is Self-Similarity Important? Recently, some network packet traffic has been identified as being self-similar Current network traffic modeling using Poisson distributing (etc.) does not take into account the self-similar nature of traffic This leads to inaccurate modeling of network traffic Is self-similarity relevant everytime?  remains a hot research area!

Courtesy of Ashish Gupta (cs.northwestern.edu) 11 Problems with Current Models A Poisson process  When observed on a fine time scale will appear bursty  When aggregated on a coarse time scale will flatten (smooth) to white noise A Self-Similar (fractal) process  When aggregated over wide range of time scales will maintain its bursty characteristic

Courtesy of Ashish Gupta (cs.northwestern.edu) 12 Pictorial View of Current Modeling

Courtesy of Ashish Gupta (cs.northwestern.edu) 13 Consequences of Self-Similarity Traffic has similar statistical properties at a range of timescales: ms, secs, mins, hrs, days Merging of traffic (as in a statistical multiplexer) does NOT result in smoothing of traffic Bursty Data Streams Aggregation Bursty Aggregate Streams

Courtesy of Ashish Gupta (cs.northwestern.edu) 14 Side-by-side View

Courtesy of Ashish Gupta (cs.northwestern.edu) 15 Definitions and Properties Long-Range Dependence  Autocorrelation {R x (t 1,t 2 ) = E[X(t 1 )X(t 2 )]} decays slowly Hurst Parameter  Developed by Harold Hurst (1965) Studies of Nile River flooding over 800 year period  H is a measure of “burstiness” also considered a measure of self-similarity  0.5 < H < 1.0

Courtesy of Ashish Gupta (cs.northwestern.edu) 16 Continuous-Time Definition Hurst Parameter The process x(t) is self-similar with parameter H if it has the same statistical properties as the process a -H x(at) for any real a>0.

Courtesy of Ashish Gupta (cs.northwestern.edu) 17 Discrete-Time Definition X = (X t : t = 0, 1, 2, ….) is random process defined at discrete points in time Let X (m) ={X k (m) } denote the new process obtained by averaging the original series X in non-overlapping sub-blocks of size m. E.g. X(1)= 4,12,34,2,-6,18,21,35 Then X(2)=8,18,6,28 X(4)=13,17

Courtesy of Ashish Gupta (cs.northwestern.edu) 18 Auto-correlation Definition X is exactly self-similar if  The aggregated processes have the same autocorrelation structure as X. i.e.  r (m) (k) = r(k), k  0 for all m =1,2, … X is asymptotically self-similar if the above holds when [ r (m) (k)  r(k), m  

Courtesy of Ashish Gupta (cs.northwestern.edu) 19 Self-Similarity in Traffic Measurement ( Ⅱ ) Network Traffic

Courtesy of Ashish Gupta (cs.northwestern.edu) 20 Auto-correlation Most striking feature of self-similarity: Correlation structures of the aggregated process do not degenerate as m   This is in contrast to traditional models  Correlation structures of their aggregated processes degenerate  i.e. r (m) (k)  0 as m  for k = 1,2,3,...

Courtesy of Ashish Gupta (cs.northwestern.edu) 21

Courtesy of Ashish Gupta (cs.northwestern.edu) 22 Long Range Dependence Processes with Long Range Dependence are characterized by an autocorrelation function that decays hyperbolically as k increases Important Property: This is also called non-summability of correlation

Courtesy of Ashish Gupta (cs.northwestern.edu) 23 Recap Self-similarity manifests itself in several equivalent fashions:  Non-degenerate autocorrelations  Slowly decaying variance  Long range dependence  Hurst effect

Courtesy of Ashish Gupta (cs.northwestern.edu) 24 The Famous Data Leland and Wilson collected hundreds of millions of Ethernet packets without loss and with recorded time-stamps accurate to within 100µs. Data collected from several Ethernet LAN’s at the Bellcore Morristown Research and Engineering Center at different times over the course of approximately 4 years.

Courtesy of Ashish Gupta (cs.northwestern.edu) 25 Plots Showing Self-Similarity ( Ⅱ ) Higher Traffic, Higher H! High Traffic Mid Traffic Low Traffic 1.3%-10.4% 3.4%-18.4% 5.0%-30.7%

Courtesy of Ashish Gupta (cs.northwestern.edu) 26 Crucial Findings Ethernet LAN traffic is statistically self-similar H  : the degree of self-similarity  H  : a function of utilization  H  : a measure of “burstiness”  Models like Poisson are not able to capture self- similarity As number of Ethernet users increases, the resulting aggregate traffic becomes burstier instead of smoother!!

Courtesy of Ashish Gupta (cs.northwestern.edu) 27 Discussions How to explain self-similarity ?  Heavy tailed file sizes How this would impact existing performance?  Limited effectiveness of buffering  Effectiveness of FEC error control for data transmission, whereby the sender adds redundant data to its messages, which allows the receiver to detect and correct errors without the need to ask the sender error controldata transmissionredundant data

Courtesy of Ashish Gupta (cs.northwestern.edu) 28 Explaining Self-Similarity The superposition of many ON/OFF sources whose ON-periods and OFF-periods exhibit the Noah Effect produces aggregate network traffic that features the Joseph Effect Also known as packet train models Noah Effect: High variability or infinite variance Joseph Effect: Self-similar or long-range dependent traffic

Courtesy of Ashish Gupta (cs.northwestern.edu) 29 The Noah Effect Noah Effect is the essential point of departure from traditional to self-similar traffic modeling Results in highly variable ON-OFF periods : Train length and inter-train distances can be very large with non-negligible probabilities Infinite Variance Syndrome : Many naturally occurring phenomenon can be well described with infinite variance distributions Heavy-tail distributions,  parameter

Courtesy of Ashish Gupta (cs.northwestern.edu) 30 Traditional Models Traditional traffic models: finite variance ON/OFF source models Superposition of such sources behaves like white noise, with only short range correlations

Courtesy of Ashish Gupta (cs.northwestern.edu) 31 The heavy-tail distribution A distribution is said to be heavy-tailed if: Property (1) is the infinite variance syndrome or the Noah Effect.   2 implies E(U 2 ) =   > 1 ensures that E(U) <  The asymptotic shape of the distribution is hyperbolic The simplest heavy-tail distribution is the Pareto distribution For example, we consider the sizes of files transferred from a web-server  Heavy-tail  A large number of small files transferred but, crucially, the number of very large files transferred remains significant.

Courtesy of Ashish Gupta (cs.northwestern.edu) 32

Courtesy of Ashish Gupta (cs.northwestern.edu) 33 Important Findings Most surprising result: Noah Effect is extremely widespread, regardless of source machine (fileserver or client machine) Explanations:  Hyperbolic tail behavior for file sizes residing in file sizes  Pareto-like tail behavior for UNIX processes run time  Human-computer interactions occur over a wide range of timescales Although network traffic is intrinsically complex, parsimonious modeling is still possible.  Estimating a single parameter  (intensity of the Noah Effect) is enough

Courtesy of Ashish Gupta (cs.northwestern.edu) 34 An example File size Distribution on a Win2000 machine

Courtesy of Ashish Gupta (cs.northwestern.edu) 35 Impact of Self Similarity

Courtesy of Ashish Gupta (cs.northwestern.edu) 36 Conclusion The presence of the Noah Effect in measured Ethernet LAN traffic is confirmed The superposition of many ON/OFF models with Noah Effect results in aggregate packet streams that are consistent with measured network traffic, and exhibits the self-similar or fractal properties Self-similarity in packetised data networks caused by the distribution of file sizes, human interactions and/or Ethernet dynamics Spawned research around the network community

Courtesy of Ashish Gupta (cs.northwestern.edu) 37 Self-similarity and long range dependence in networks Vern Paxson and Sally Floyd, Wide-Area Traffic: The Failure of Poisson ModelingWide-Area Traffic: The Failure of Poisson Modeling Mark E. Crovella and Azer Bestavros, Self-Similarity in World Wide Web Traffic: Evidence and Possible CausesSelf-Similarity in World Wide Web Traffic: Evidence and Possible Causes  It shows that self-similarity in Web traffic can be explained based on the underlying distribution of transferred document sizes, the effects of caching and user preference in file transfer, the effect of user ``think time'', and the superimposition of many such transfers in a local area network. A. Feldmann, A. C. Gilbert, W. Willinger, and T. G. Kurtz, The Changing Nature of Network Traffic: Scaling Phenomena,The Changing Nature of Network Traffic: Scaling Phenomena Mark Garrett and Walter Willinger, Analysis, Modeling and Generation of Self- Similar VBR Video TrafficAnalysis, Modeling and Generation of Self- Similar VBR Video Traffic  The paper shows that the marginal bandwidth distribution can be described as being heavy- tailed and that the video sequence itself is long-range dependent and can be modeled using a self-similar process  The paper presents a new source model for VBR video traffic and describes how it may be used to generate VBR traffic synthetically.

Courtesy of Ashish Gupta (cs.northwestern.edu) 38 Heavy tailed distributions in network traffic Gordon Irlam, Unix File Size Survey,Unix File Size Survey Will Leland and Teun Ott, Load-balancing Heuristics and Process Behavior, Mor Harchol-Balter and Allen Downey, Exploiting Process Lifetime Distributions for Dynamic Load BalancingExploiting Process Lifetime Distributions for Dynamic Load Balancing Carlos Cunha, Azer Bestavros, Mark Crovella, Characteristics of WWW Client-based TracesCharacteristics of WWW Client-based Traces  This paper presents some of the first Web client measurement ever made. It characterizes traces taken using an instrumented version of Mosaic from a university computer lab and shows that a number of Web properties can be modeled using heavy tailed distributions.  These properties include document size, user requests for a document, and document popularity.