New Directions in Traffic Measurement and Accounting Cristian Estan (joint work with George Varghese)

The Problem Measurement and monitoring of network traffic required for Internet backbones. Useful for short-term monitoring (e.g., DOS attacks), traffic engineering (e.g., rerouting), and accounting (e.g., usage based pricing) How can we do so without tracking millions of ants to track a few elephants?

State of the art – Cisco NetFlow Sample packets at high speeds; Per flow information based on samples; Aggregate (based on ASes, prefixes, ports) at the router; Problems: inaccurate (due to sampling and loss), memory-intensive, slow (needs DRAM).

Towards a NetFlow Alternative Small Percentage of flows (elephants) account for large percentage of traffic. Top 9% of flows account for 90% of AS pair traffic in backbones (Fang-Peterson). Can we directly track flows that send say over 1% of link bandwidth without keeping track of all flows?

How to identify large flows? Sample-Counting: uses sampling only to decide which flows to watch exhaustively. Multistage filter: uses multiple hash tables allowing large flows to be identified while only allowing a small number of small flows (false positives) to pass through filter. We introduce two new methods for this purpose:

Identify large flows by sampling

Multistage filters

Operation of Sampled NetFlow How accurate is Sampled NetFlow? 1 gigabyte/100 megabytes of data Sampling one in 100 packets Error1GB100MB 1%39.24%79.03% 3%1.07%41.48%

Operation of our algorithms Error1GB100MB 1%5.6E % 3%1.8E E-10 Error1GB100MB 1%0.08%49.69% 3%4.66E % Sampling 1/1000 Filter error: 0%

Comparison Sampled NetFlow Identify by sampling Multistage filters AccuracyMediumGood False neg.Few (high var.) Few (low var.) Never False pos.Few MemoryBigSmallVery small ComplexityLow Medium

High Speed Implementation? John Huber of MMC Networks did a design of a chip doing filter counting. 450,000 transistors, under 1 watt of power, runs at OC-192 rates, 32 nsec per packet Seems easily feasible to implement sample counting with similar complexity.

Potential Application: scalable threshold accounting Measure flows sending over x% of link bandwidth using sample/filter counting. Bill using flat fee + per byte charge for flows over x% Track aggregates directly to avoid evasion using several flows, each < x% Generalizes usage based (x = 0) and duration based (x = 100) pricing.

Conclusions Paradigm shift for measurement by concentrating only on heavy flows Two new techniques (sample and filter counting) with small memory footprints and provable performance. Techniques make threshold accounting feasible, generalizing usage and duration based pricing.