1. Enabling a “RISC” Approach for Software-Defined Monitoring using Universal Streaming Vyas Sekar Zaoxing Liu, Greg Vorsanger, Vladimir Braverman

2. Network Management: Many Monitoring Requirements SDN Controller (OpenDayLight etc.) Traffic Engineering Analyze new user apps Anomaly Detection NetworkForensics Worm Detection Accounting Botnet analysis ……. “Heavy-hitters” “Flow size distribution” “SuperSpreaders” “Entropy”, “Traffic Changes” 1

3. Traditional: Packet Sampling 2 161311 1 Flow reports 1 Not good for fine-grained analysis Extensive literature on limitations for many tasks! 11316111131611 12 Sample packets at random, aggregate into flows FlowId Counter Flow = Packets with same pattern Source and Destination Address and Ports Estimate: FSD, Entropy, Heavyhitters, Changes, SuperSpreaders ….

4. Application-Specific Sketches Packet Processing Counter Data Structures Application-Level Metric Heavy Hitter EntropySuperspreader Complexity: Need per-metric implementation Recent Example: OpenSketch [NSDI’13] Trend: Many more applications appear! …. Monitoring (on router) Bloom-filter, Count-min Sketch, reversible sketch, etc. 3 Packet Processing Counter Data Structures Application-Level Metric Packet Processing Counter Data Structures Application-Level Metric …. Traffic Computation (off router)

5. Packet Processing Counter Data Structures Application-Level Metric Support many applications Holy Grail of Flow Monitoring? Results with high accuracy 4 Traffic

6. Our Solution: Universal Monitoring 5 Recent theory advances: Universal Streaming Packet Processing Universal Sketch Traffic App 1 Application-specific Computation App n …... UnivMon Control Plane UnivMon Data Plane One sketch does it ALL

7. Theory of Universal Streaming 1. Vladimir Braverman, Rafail Ostrovsky: Zero-one frequency laws. STOC 2010 2. Generalizing the Layering Method of Indyk and Woodruff: Recursive Sketches for Frequency-Based Vectors on Streams. APPROX-RANDOM 2013 13315112465 …... (A stream of length m with n unique items) ‘Universal’ Sketch Estimated G-sum frequency vector is 6

8. Universal Sketch Data Structure 13315112465 11511 25 2 L2 Heavy Hitter Algorithms (1,4), (3,2),(5,2) Heavy Hitters (1,4), (5,2),(2,1) …... (2,1) 7 (5,2), (2,1) 0 1 log(n) …... Generate k=log(n) pairwise ind. zero-one hash functions: H 1 …. H k 25 5 Similar to counting bloom filter H 1 (1)=1, H 1 (5)=1, H 1 (2)=1 H 2 (5)=1, H 2 (2)=1 H 3 (2)=1 Levels Heavy Hitter Alg Count Sketch Alg +4+2-2 -4+2 +4-2 +4-2 +4-2+1 -2-4 +1 +1 …... Count-Sketch, Pick-and- drop etc. In Parallel

9. Estimating G-sum (1,4), (3,2),(5,2) Counters from Universal Sketch (1,4), (5,2), (2,1) …... (2,1) 8 (5,2),(2,1) Levels 0 1 log(n) …... Apply arbitrary g() (1,g(4)), (3,g(2)),(5,g(2)) (1,g(4)), (5,g(2)), (2,g(1)) (5,g(2)),(2,g(1)) (2,g(1)) Y 3 =g(1) Sum of the g()s Y 2 =g(1)+g(2) Y 1 =g(1)+g(2)+g(4) Y 0 =2g(1)+2g(2)+g(4) Estimated G-sum Recursive Steps: Y i-1 = 2Y i + new counters – repeated counters

10. Putting it together: UnivMon Universal Sketch Offline Recursive Computation 9

11. Comparison with custom sketches via OpenSketch Preliminary Evaluation 10 N/A

12. Distributed universal streaming Multidimensional data Dynamically change monitoring scope Feasibility of hardware implementations? Future Directions 11

13. 12 Conclusions Network management needs many traffic metrics Today’s solutions offer undesirable extremes Generic but low fidelity (e.g., sampling) High fidelity but high complexity (e.g., specific-sketches) Holy grail: Universal Monitoring Decouple monitoring control and data plane like SDN! This work: Can be viable via Universal Sketches Several open questions e.g. dynamic, multidimensional, distributed, hardware viability