collaborators: Mark Coates, Rui Castro, Ryan King, Mike Rabbat, Yolanda Tsang, Vinay Ribeiro, Shri Sarvotham, Rolf Reidi Network Bandwidth Estimation and Tomography Rob Nowak & Rich Baraniuk UW-Madison Rice University spin.rice.edu

too complex to measure everywhere, all the time traffic measurements expensive (hardware, bandwidth) Internet Boom

companies do not share data or performance information Proprietary Concerns

bits are bundled into packets packets go through routers queues absorb bursts of packets congestion: queues fill up, large delays, packet drops Networking 101

Network Measurement & Inference Internet equivalent model Path Modeling and Bandwidth Estimation Network Tomography

Brain Tomography unknown object statistical model measurements Maximum likelihood estimate maximize likelihood physics data prior knowledge MRF model counting & projection Poisson

unknown object statistical model measurements Maximum likelihood estimate maximize likelihood physics data prior knowledge Network Tomography queuing behavior routing & counting binomial / multinomial

Network Tomography From link-level traffic measurements, infer end-to-end traffic flow rates Vardi 96, Tebaldi & West 98 Cao, Davis, Vander Wiel, Yu 00

y = packet losses or delays measured at the edge A = routing matrix (graph) = packet loss probabilities or queuing delays for each link = randomness inherent traffic measurements likelihood function Network Tomography (MINC Project, Towsley-Duffield)

Probe packets experience similar queuing effects and may interact with each other Probing the Network probe = packet stripe cross-traffic delay

Network Tomography: The Basic Idea sender receivers

Network Tomography: The Basic Idea sender receivers

Maximum Likelihood Estimation via EM Suppose we were able to measure losses/delays on each link Expectation-Maximization (EM) alternates between computing expectation of unobserved internal measurements and the desired estimates of link-by-link loss/delay distributions Problem: How to compute maximum likelihood estimates of link-by-link loss/delay distributions from end-to-end measurements ?

Topology ID via Probe Interactions

we can infer that receivers 3 & 4 have a longer shared path than 3 & 5

Finding the Maximum Likelihood Tree Stochastic search through forest via Metropolis-Hastings

True topology estimated topology Internet measurement experiments UNO

What have we learned? Clever probing and sampling schemes reveal hidden network structure and behavior Simple inference algorithms are effective, intuitive, easy to implement, scale nicely MLE criteria are easily modified to include prior information: Bayesian or regularized MLE methods are straightforward Complex interplay between measurement/probing techniques, statistical modeling, and computational methods for optimization

Open Problems: Placement/Coverage How should measurement devices be deployed ? Logical graph coverage of physical topology ? Can random graph models shed some light ?

Open Problems: Spatio-temporal Correlation competing traffic Can we detect correlations? Can we exploit them in measurement and mapping applications? Fuse tomography and bandwidth estimation Long-range dependence of network traffic Correlations due to competing traffic flows

Open Problems: Detection and Localization Detecting and locating anomalous behavior rather than estimating everything EstimationHypothesis Testing How can we capitalize on conventional wisdom: most links are good and only a few are bad ?

Open Problems: Timing and Synchronization Hardware solutions (expensive) Software solutions (more practical) - sophisticated software clocks (Veitch 02) - crude software clocks (ICMP timestamping) and statistical averaging sender network sender monitor receiver monitor receiver How to accurately measure time ?

Open Problems: Network Security How can measurement and monitoring across the Internet help detect and prevent malicious activities ?