Scalable and Deterministic Overlay Network Diagnosis Yao Zhao, Yan Chen Northwestern Lab for Internet and Security Technology (LIST) Dept. of Computer Science Northwestern University David Bindel Computer Science Division Dept. of EECS University of California at Berkeley

When something breaks in the Internet, the Internet's very decentralized structure makes it hard to figure out what went wrong and even harder to assign responsibility. ̶̶ ̶ “Looking Over the Fence at Networks: A Neighbor's View of Networking Research”, by Committees on Research Horizons in Networking, National Research Council, 2001.

Motivation Internet diagnosis very important To end users To overlay network service providers (e.g., Akamai) To Internet service providers (ISP) But a very challenging problem due to the privacy of network administration Solution E2E measurements by end users -- overlay networks

Related Works Router based approaches [SOSP03] Mostly ICMP based, ICMP rate limiting Unscalable for simultaneous diagnosis Cannot deterministically separate forward/backward path loss Statistical approaches [MINC, INFOCOM03] Non-deterministic: fundamentally under-constrained system Inference based on temporal correlation in a multicast tree Have to compromise for unicast, then sensitive to cross traffic Optimization based on assumptions: # of lossy links small Random sampling, linear programming, and Bayesian inference. Unscalable: iterative refinement slow to converge for large networks

Problem Formulation Given an overlay of N end hosts and O(N 2 ) paths, to what granularity can we deterministically diagnosis the network fault? Assumptions: Topology measurable Can only measure the E2E path, not the link

Outlines Architecture and algebraic model Identifying virtual links Evaluation with simulations Internet experiments

Our Approach Monitor a basis set of O(n·logn) paths that fully describe the O(n 2 ) paths Decompose the paths into minimal deterministically identifiable segments Compute the loss rate for each segment for diagnosis End hosts Overlay Network Operation Center topology measurements Trouble spots location Diagnosis results: Qwest access link: > Peering between UUNET and AOL: >

Linear algebraic model Path loss rate p, link loss rate l D C B p1p1

Putting All Paths Together … =

Identifiable and Unidentifiable Vectors in the row space of G are identifiable Otherwise, unidentifiable (1,-1,0) (1,1,0) Row(path) space (identifiable) x1x1 x3x3 A D C B p1p1 p2p2 (1,1,1) (0,0,1) x2x2

Outlines Architecture and algebraic model Identifying virtual links Evaluation with simulations Internet experiments

Definition of Virtual Links Uniquely identified shortest path segments Identifiable Consecutive Undecomposable ’ 2’ 3’ 4’ 4 5 a b d ce 4 paths, 5 links5 virtual links

One More Example 6 paths, 8 links 4 virtual links: Corresponding to links 1, 2, and respectively ’ 3’ 6’ 4 5’ ’ 4’

Computing Virtual Links in Undirected Graph (1,0,0) (1,1,0) Row space x1x1 x2x2 (1,1,1) (0,1,0) x3x3 Check if a vector is a virtual link QR decomposition: O(l·k) to check if a vector of length l is in row space of G O(l 2 ) potential virtual links in a path of length l Total complexity O(l·k·l 2 ·k)=O(l 3 ·k 2 ) Small constant: only 4.2 sec for 135-node network

Undirected vs. Directed Graphs Directed graph Any linear combination => Theorem: In a directed graph, no end-to-end path contains an identifiable subpath.

Rescue: Good Path Algorithm Identifying virtual links in undirected graphs Use topology only For directed graphs: additional info needed Path loss rate Use the link property constraint to break the deadlock All the links in a good path are good links, i.e. no or little loss. Most of the paths on the Internet are good paths

System Flowchart Monitors O(n·logn) paths that can fully describe all the O(n 2 ) paths (SIGCOMM04) Inherit load balancing, monitoring adaptation, etc. Measure topology to get G Select a basis of G,, for monitoring Good path algorithm on Reduced paths G’ Reduced paths G’’ Select a basis of G’’: Find all lossy virtual links in G Estimated loss rates for all paths in G Good path algorithm on G Stage 2: online update the measurements and diagnosis Stage 1: set up scalable monitoring system for diagnosis Optimization steps: find the minimal basis for identifiability test

Outlines Architecture and algebraic model Identifying virtual links Evaluation with simulations Internet experiments

Metrics Avg length of lossy virtual links in all lossy paths Diagnosis granularity The avg number of potential lossy links in a lossy path Example (Path 1 w/ lossy VL 1 of length 5, path 2 and 3 w/ lossy VL 2 of length 2) Avg lossy VL length: (5+2)/2 = 3.5 Avg diagnosis granularity: (5+2+2)/3 = 3 Accuracy Absolute error |p – p’ | Relative error

Simulation Methodology Topology type Three types of BRITE router-level topologies Mecator topology Topology size 1000 ~ or 184k nodes Fraction of end hosts on the overlay network 10% ~ 50% Link loss rate distribution LLRD 1 and LLRD 2 models Loss model Bernoulli and Gilbert

Sample of Simulation Results (Barabasi+Gilbert)

Results using Mercator Topology # of end hosts on OL Avg LP # of LP # of links in LP Avg LP Length Avg VLL in LP Avg BVLL in LP (4.86)2.56(3.54)2.97(4.18) (4.5)1.76(2.36)2.21(3.11) (4.21)1.6(2.07)1.99(2.74)

Gibbs Sampling (Infocom03) D Observed packet transmission and loss at the clients  Ensemble of loss rates of links in the network Goal Determine the posterior distribution P(  |D) Approach Use Markov Chain Monte Carlo with Gibbs sampling to obtain samples from P(  |D) Draw conclusions based on the samples

Comparison with Bayesian Inference using Gibbs Sampling (1)

Comparison with Bayesian Inference using Gibbs Sampling (2)

Outlines Architecture and algebraic model Identifying virtual links Evaluation with simulations Internet experiments

Methodology Planetlab 135 end hosts Topology measured by Traceroute Avg path length is 17.2 Path loss rate by active UDP probing byte UDP packets per measured path in 90 sec Small overhead: 17.9kb if even measuring all paths

Diagnosis Results Total end-to-end paths18,090 Avg Path Length17.2 After removing 79.5% good paths w/ 80.5% good links … Avg lossy path (>5% loss rate) length11.5 (9.0) Avg lossy virtual link length4.3 (3.1) Avg Granularity4.0 (2.7) Loss rate [0, 0.05) lossy path [0.05, 1.0] (15.8%) [0.05, 0.1)[0.1, 0.3)[0.3, 0.5)[0.5, 1.0)1.0 % The numbers in () are those after removing sequential link chains.

Speed Results On a Pentium-IV 3.2GHz PC Average setup time (selecting 5,706 paths for monitoring): seconds Diagnosis of 2,858 lossy paths: 4.2 seconds

Validation Cross Validation Divide 5720 paths into two sets (2860 each) Get 571 virtual links from the first set Check consistency with the second path set 99.1% paths in the second set are consistent with virtual links computed by the first set.

IP Spoofing based Validation UDP: S:a, D:c, TTL=255 a c b UDP: S:a, D:b, TTL=255 UDP: S:c, D:b, TTL=2 ICMP: S:r3, D:c, TTL=255 r1r1 r2r2 r3r3

IP Spoofing based Consistency Checking Use the function of source routing of IP Spoofing to create new path segments Validation is the same as cross validation Results: 1000 new path including part of segments in potential lossy paths 94.1% loss spoofed paths are consistent with 361 out of 1664 lossy virtual links 5.9% paths are inconsistent with 45 virtual links

Conclusions Propose the first deterministic and scalable overlay diagnosis system based on a linear algebraic approach Diagnosis with virtual links: Identifiable, consecutive and minimal path segments Directed topology indecomposable to VL Good path algorithms for rescue Both simulation and Internet experiments show fast & accurate diagnosis w/ optimal granularity

Backup Slides

Previous Work “Computing the unmeasured: An algebraic approach to Internet mapping,” INFOCOM’01 Can’t work on directed graph “User-level internet path diagnosis,” SOSP’03 Need the support of routers Not accurate “Multicast-based inference of network-internal loss characteristics,” IEEE Transactions in Information Theory, Multicast support or unicast approximation “Server-based inference of Internet link lossiness,” INFOCOM'03 Can only determine whether a link is lossy or not

Distribution of Length of lossy Virtual Links

IP Spoof Based Diagnosis