1 Network Measurements and Sampling Nick Duffield, Carsten Lund, and Mikkel Thorup AT&T Labs-Research, Florham Park, NJ

2 IP Network The Edge of Network Links Routers

3 Some Questions that are hard to answer about IP networks o Given a hotspot, how do we change network configuration to fix the problem. ¨ What exactly is the problem? Type of traffic? o Traffic usage. Which applications are using the network? o How do we design an efficient network? ¨ Where do the traffic want to go?

4 Traffic Matrix Construction o Traffic Matrix ¨ Point to Point Demands ¨ Point to Multi-Point Demands –Needed to model in networks with multi-homing and peering. (e.g., ISPs etc.) o Input for many tasks ¨ Network Design ¨ Reroute traffic ¨ Failure analysis ¨ OSFP weight optimization (Thorup et al.)

5 Measurement Options o SNMP ¨ Counting packets on interfaces o Packet sniffers/Probes ¨ Whole packet header/contents o Flow measurements ¨ Router support ¨ Network-wide measurements o Packet sampling (sFlow) o Trajectory sampling o Etc.

6 Traffic Matrix: SNMP based methods o Network Tomography ¨ From link measurement determine traffic matrix, using temporal correlations. o Network Tomogravity ¨ Starts with a model of the traffic matrix from edge measurements. ¨ Uses internal measurements to find a consistent solutions that is close to the traffic matrix that the model predicted.

7 Traffic Matrix: Flow based methods. o Detailed measurement where the traffic enter and determine from the destination IP address, where the traffic will exits the network ¨ Need Prefix level routing information to determine the exit point

8 Traffic Matrix: Method Comparison o SNMP based ¨ Less accurate ¨ Harder to deal with multi-point destinations –Tomography do not. –Tomogravity do try to model it. ¨ Measurement instrumentation/gathering is standard. o Flow based ¨ More accurate ¨ Deal with the multi-point destinations ¨ Measurement instrumentation/gathering is not standard. ¨ Need to deal with huge amount of detailed measurements

9 Outline of the rest of the talk o Context ¨ What are Flows? ¨ Measurement Infrastructure o The main idea ¨ Size dependent sampling ¨ Optimal sampling o How do this compare to size independent sampling? o Billing application

10 flow 1flow 2flow 3 flow 4 IP Flow Abstraction o IP flow abstraction ¨ set of packets identified with “same” address, ports, etc. ¨ packets that are “close” together in time o IP flow summaries ¨ reports of measured flows from routers –flow identifiers, total packets/bytes, start and end times o Several flow definitions in commercial use

11 Collection System Flow Data Aggregate Data Routers Flow Collectors Central Server

12 Router CPU RAM Collectors CPU RAM Central Server Disk CPU RAM Bandwidth Resources in the Measurement System

13 Measure/Export All Traffic Flows? o Flow volumes ¨ one OC48 (2.4 Gbps Trunk)  several GB flow summaries per hour o Cost ¨ network resources for transmission ¨ storage/processing at the collection system

14 Flow Sampling? o Sampling ¨ statisticians reflex action to large datasets o Export selected flows ¨ reduce transmission/storage/processing costs o Sufficiently accurate? ¨ Depends on application. –Traffic Engineering (where do traffic flow) –Traffic Analysis (application usage, etc.) –Billing l risk of overcharging (  irate customers) l risk of undercharging (  irate shareholders)

15 Packet Sampling and Flow Sampling o Packet Sampling ¨ when router can’t form flows at line rate –scaling at a single router o Flow sampling ¨ managing volume of flow statistics –scaling across downstream measurement infrastructure o Complementary ¨ could combine –e.g. 1 in N packet sampling + flow sampling

16 Usage Estimation o Each flow i has ¨ “size” x i –bytes or packets ¨ “color” c i –combination of IP address, port, ToS etc that maps to billable stream ( = customer + billing class) o Goal ¨ to estimate total usage X(c) in each color c

17 Basic Ideas o Match sampling method to flow characteristics ¨ high fraction of traffic found in small fraction of long flows –sample long flows more frequently than short flows l large contributions to usage more reliably estimated o Show how to relate sampling and accuracy ¨ simple rules to achieve desired accuracy

18 Size independent flow sampling bad o Sample 1 in N flows ¨ estimate total bytes by N times sampled bytes o Problem: ¨ long flow lengths –estimate sensitive to inclusion or omission of a single large flow

19 Size dependent flow sampling o Sample flow summary of size x with prob. p(x) o Estimate usage X by ¨ boost up size x by factor 1/p(x) in estimate X’ –compensate against chance of being sampled o Chose p(x) to be increasing in x ¨ longer flows more likely to be sampled ¨ compare size independent sampling: p(x) =1/N

20 Statistical Properties o Fixed set of flow sizes {x 1, x 2, …,x n } ¨ we only consider randomness of sampling o X’ is unbiased estimator of actual usage X =  i x i ¨  X’ = X: averaging over all possible samplings ¨ holds for all probability functions p(x) o Proof: ¨ X’ =  i w i x i /p(x i ) –w i random variable l w i =1 with prob. p(x i ), 0 otherwise –  w i = p(x i ) hence  X’ =   i w i x i /p(x i )=  i x i =X

21 What is best choice of p(x)? o Trade-off accuracy vs. number of samples o Express trade-off through cost function ¨ cost = variance(X’) + z 2 average number of samples –parameter z: relative importance of variance vs. # samples o Which choice of p(x) minimizes cost? o p z (x) = min { 1, x/z } ¨ flows with size  z: always selected ¨ flows with size < z: selected with prob. proportional to their size o Trade-off ¨ smaller z –more samples, lower variance ¨ larger z –fewer samples, higher variance o Will call sampling with p z (x) “optimal” p z (x) z 1 x

22 Optimal vs. size independent sampling o NetFlow traces ¨ 1000’s cable users, 1 week o Color flows ¨ by customer-side IP address c o Compare ¨ 1 in N sampling ¨ optimal sampling –same average sampling rate o Measure of accuracy ¨ weighted mean relative error o Heavy tailed flow size distribution is our friend! ¨ allows more accurate encoding of usage information

23 Charging and Sampling Error o Optimal sampling ¨ no sampling error for flows larger than z o Exploit in charging scheme ¨ fixed charge for small usage ¨ usage sensitive charge only for usage above insensitivity level L o Charge according to estimated usage f(X’(c)) = a + b max{ L, X’(c) } –coefficients a, b and level L could depend on color c o Only usage above L needs reliable estimation f(X) X L

24 Accuracy and Parameter Choice o Given target accuracy ¨ relate sampling threshold z to level L o Theorem ¨ Variance(X’)  z X (tight bound) ¨ now assume: z   2 L –Std.Dev. X’   X if X  L l bound sampling error of estimated usage > L –Std.Dev. f(X’)   f(X) l bound error of charge based on estimated usage o Bounds hold for any flow sizes {x i } ¨ no assumption on flow size distribution –just choose z   2 L

25 Example o Target parameters ¨ L = 10 7,  = 10%  z = 10 5 o Scatter plot ¨ ratio estimated/actual usage vs. actual usage –each color c ¨ observe better estimation of higher usage o Want to avoid ¨ ratio > 1+  = 1.1 and usage > L = 10 7 o Less than 1 in 1000 “bad” points

26 o Aim: ¨ reduce chance of overestimating usage o Method: ¨ theorem gave bound: Var(X’)  z X ¨ anticipate upwards variations in X’ by subtracting off multiples of std. dev. –charge according to ¨ again: no assumptions on flow size distribution Compensating variance for mean

27 Example: s=1 o Scatter pushed down: ¨ no points with ratio>1.1 and usage > 10 7 o Drawback ¨ more unbillable usage –when X’ s <X o Small unbillable usage for heavy users ¨ ratio  1 ¨ Std.Dev.(X’)/X’ vanishes as X grows

28 Example: s=2 o Scatter pushed down further: ¨ no points with ratio > 1 o Trade off ¨ unbillable usage vs. overestimation sunbill. bytes X’ s >X? 0-0.1%50% 13.1%3% 26.2%0%

29 How to reduce unbillable usage? o Make sampling more accurate ¨ reduce z! o For unbillable fraction <  ¨ chose s z   2 L o Example: ¨ s = 2,  = 10% ¨ reduce z –from 10 5 to 10 4 o Alternative ¨ increase coefficent a in charge f(X) to cover costs

30 Tension between accuracy and volume o Want to reduce z ¨ better accuracy, less unbillable usage o Drawback ¨ increased sample volume o Solution ¨ make billing period longer instead –usage roughly proportional to billing period –allows increased charge insensitivity level L ¨ sampling production rate controlled by threshold z –rate r  x f(x)p z (x) l flow arrival rate r, fraction f(x) of flows size x o Need only z =  2 L ¨ larger L allows smaller error  for given z

31 Summary o Size dependent optimal sampling ¨ preferentially sample large flows –more accurate usage estimates for given sample volume –sample flow of size x with probability p z (x) o Charging from measured usage X’ ¨ charge f(X’) = a +b max{L,X’} –fixed charge for usage below insensitivity level L –only need to reliably estimate usage above L o Sampling/charging accuracy ¨ choose z =  2 L to get standard error  o Variance compensation ¨ replace X’ by o Longer billing cycle ¨ increases L, better accuracy (  ) at given sampling rate (z)

32 Papers o