Data Streaming in Computer Networking Cristian Estan, George Varghese University of California, San Diego

Data streaming in computer networking - MPDS 2003 Talk structure Traditional streaming in networking Rules of the game Iteration paradigm: packet scheduling example New streaming problems Detecting malicious traffic Understanding network workloads June 8, 2003 Data streaming in computer networking - MPDS 2003

Internet service model Source port Destination port Source IP address Destination IP address Data Header Conversations (flows) broken up into packets handled independently by the network Packets contain detailed information Destination IP address Source IP address “Application”: protocol field + source and destination port At the core of the network high speed routers Decide what to do with each packet Flow Internet June 8, 2003 Data streaming in computer networking - MPDS 2003

Traditional router functions IP Lookup ? Incoming 1 Outgoing 1 Incoming 2 Outgoing 2 Decide which interface to send the packet on (route lookup) Incoming 3 Outgoing 3 June 8, 2003 Data streaming in computer networking - MPDS 2003

Traditional router functions IP Lookup Out2 Incoming 1 Outgoing 1 Incoming 2 Outgoing 2 Incoming 3 Outgoing 3 June 8, 2003 Data streaming in computer networking - MPDS 2003

Traditional router functions Switching Out2 Out3 Incoming 1 Outgoing 1 Out3 Incoming 2 Outgoing 2 Move packets from between interfaces (switching) Out1 Out2 Incoming 3 Outgoing 3 June 8, 2003 Data streaming in computer networking - MPDS 2003

Traditional router functions Scheduling Incoming 1 Outgoing 1 Flow 1 Flow 2 Incoming 2 Outgoing 2 Flow 3 Decide which packets to send and which to delay or drop (scheduling) Incoming 3 Outgoing 3 June 8, 2003 Data streaming in computer networking - MPDS 2003

Traditional router functions Scheduling Incoming 1 Outgoing 1 Flow 1 Flow 3 Flow 2 Incoming 2 Outgoing 2 Incoming 3 Outgoing 3 June 8, 2003 Data streaming in computer networking - MPDS 2003

Data streaming in computer networking - MPDS 2003 Rules of the game Wire speed processing At 40 gigabits/s 8 nanoseconds per packet - need fast SRAM Limited SRAM (say 32 megabits) but millions of flows What does this mean for algorithms? Low worst case complexity bounds Low bounds on the amount of memory used Differences from databases One pass vs. multiple passes Worst case vs. average case Small constants vs. asymptotic complexity June 8, 2003 Data streaming in computer networking - MPDS 2003

Data streaming in computer networking - MPDS 2003 Talk structure Traditional streaming in networking Rules of the game Iteration paradigm: packet scheduling example New streaming problems Detecting malicious traffic Understanding network workloads June 8, 2003 Data streaming in computer networking - MPDS 2003

Data streaming in computer networking - MPDS 2003 Iteration paradigm Many networking algorithms use iteration in time Way to allow multi-pass algorithms without storing input by assuming inputs do not change quickly Many examples (MULTOPS for DoS detection [Gil01], CSFQ for scheduling [Stoica98]) Would be nice to formalize tradeoff between quality of results and drift rate of input Perhaps exponential averaging is not enough June 8, 2003 Data streaming in computer networking - MPDS 2003

Example: Core Stateless FQ If R>F drop with probability 1-F/R Iteratively compute fair share F R Fair queuing: if traffic is larger than link capacity, limit the large flows to the “fair share” The size of the fair share depends on the rates of all flows Per flow state is impractical Core Stateless: Uses labels in packets to determine rates of flows Computes the fair share using iterative approach Minimal state Exploits stationarity in the traffic mix Mark rate R June 8, 2003 Data streaming in computer networking - MPDS 2003

Data streaming in computer networking - MPDS 2003 Talk structure Traditional streaming in networking Rules of the game Iteration paradigm: packet scheduling example New streaming problems Detecting malicious traffic Understanding network workloads June 8, 2003 Data streaming in computer networking - MPDS 2003

New streaming problems Detecting malicious activity Flooding (denial of service attacks) Worms Scans looking for vulnerable servers Understanding workloads Billing Planning network growth Application mix June 8, 2003 Data streaming in computer networking - MPDS 2003

Detecting malicious traffic Well defined building blocks Detecting large aggregates Similar to iceberg queries Counting active flows in an aggregate Similar to counting distinct values Many open problems: e.g. detect worms and DoS attacks (not clear what is right formal problem statement) June 8, 2003 Data streaming in computer networking - MPDS 2003

Data streaming in computer networking - MPDS 2003 Talk structure Traditional streaming in networking Rules of the game Iteration paradigm: packet scheduling example New streaming problems Detecting malicious traffic Understanding network workloads June 8, 2003 Data streaming in computer networking - MPDS 2003

Informal problem definition Analysis Traffic reports Applications: 50% of traffic is Kazaa Sources: 20% of traffic comes from Steve’s PC Terabytes of measurement data June 8, 2003 Data streaming in computer networking - MPDS 2003

Informal problem definition Analysis Traffic reports 20% is Kazaa from Steve’s PC 50% is Kazaa from the dorms Terabytes of measurement data June 8, 2003 Data streaming in computer networking - MPDS 2003

Formal problem definition Define clusters: Atoms: fields 1 to n with hierarchies in each field including * Cluster: intersection of one set from each field hierarchy Example: Source=*, Destination=CS Net, App= Email Threshold clusters: Report traffic clusters above threshold T (e.g. 1% of traffic) Omit redundant clusters: Compression rule: remove general clusters from report when its traffic can be inferred (up to error T) from on non-overlapping more specific clusters June 8, 2003 Data streaming in computer networking - MPDS 2003

Data streaming in computer networking - MPDS 2003 Solution status The good: Offline tool AutoFocus; SIGCOMM 2003 paper Detected worm, busy servers, squid cache, etc. Network managers like it The bad: Takes long: 3 hours at T=0.5% for one day trace Needs much memory 300 Mbytes The wanted: Streaming algorithm - we invite improvements June 8, 2003 Data streaming in computer networking - MPDS 2003

Data streaming in computer networking - MPDS 2003 Conclusions New rules: strict constraints on algorithms running in routers Iteration in time: can give simple algorithms, but needs more formalization as to quality of results General open problems: many challenges in detecting malicious traffic such as worms and DoS attacks Specific open problem: computing traffic cluster reports in streaming fashion June 8, 2003 Data streaming in computer networking - MPDS 2003

Data streaming in computer networking - MPDS 2003 Thank you! Algorithms ? Databases Networking June 8, 2003 Data streaming in computer networking - MPDS 2003

Unidimensional clusters 15 35 30 40 160 110 35 75 10.8.0.2 10.8.0.3 10.8.0.4 10.8.0.5 10.8.0.8 10.8.0.9 10.8.0.10 10.8.0.14 June 8, 2003 Data streaming in computer networking - MPDS 2003

Unidimensional clusters 10.8.0.0/28 500 10.8.0.0/29 120 10.8.0.8/29 380 10.8.0.0/30 50 10.8.0.4/30 70 10.8.0.8/30 305 75 10.8.0.12/30 10.8.0.10/31 10.8.0.2/31 50 10.8.0.4/31 70 10.8.0.8/31 270 35 75 10.8.0.14/31 15 35 30 40 160 110 35 75 10.8.0.2 10.8.0.3 10.8.0.4 10.8.0.5 10.8.0.8 10.8.0.9 10.8.0.10 10.8.0.14 June 8, 2003 Data streaming in computer networking - MPDS 2003

Unidimensional clusters 10.8.0.0/28 500 10.8.0.0/29 120 10.8.0.8/29 380 10.8.0.0/30 50 10.8.0.4/30 70 10.8.0.8/30 305 75 10.8.0.12/30 10.8.0.10/31 10.8.0.2/31 50 10.8.0.4/31 70 10.8.0.8/31 270 35 75 10.8.0.14/31 15 35 30 40 160 110 35 75 10.8.0.2 10.8.0.3 10.8.0.4 10.8.0.5 10.8.0.8 10.8.0.9 10.8.0.10 10.8.0.14 June 8, 2003 Data streaming in computer networking - MPDS 2003

Unidimensional clusters 10.8.0.0/28 500 10.8.0.0/29 120 10.8.0.8/29 380 10.8.0.8/30 305 10.8.0.8/31 270 160 110 10.8.0.8 10.8.0.9 June 8, 2003 Data streaming in computer networking - MPDS 2003

Unidimensional clusters 10.8.0.0/28 500 10.8.0.0/29 120 10.8.0.8/29 380 10.8.0.8/30 305 10.8.0.8/31 270 160 110 10.8.0.8 10.8.0.9 June 8, 2003 Data streaming in computer networking - MPDS 2003

Multidimensional clusters Two dimensions Source network Protocol (traffic type) Trees turn into lattice Multiple parents Nodes overlap June 8, 2003 Data streaming in computer networking - MPDS 2003

Data streaming in computer networking - MPDS 2003 Offline solution June 8, 2003 Data streaming in computer networking - MPDS 2003

Data streaming in computer networking - MPDS 2003 Sample report June 8, 2003 Data streaming in computer networking - MPDS 2003