Routers with Small Buffers Yashar Ganjali High Performance Networking Group Stanford University Joint work with: Guido Appenzeller, Mihaela Enachescu, Ashish Goel, Tim Roughgarden, Nick McKeown Special thanks to: Level 3 Communications NANOG, October 25, 2005

October 2005 Routers with Small Buffers 2 The Story (1)Assume: Large number of desynchronized flows; 100% utilization (2)Assume: Large number of flows; <100% utilization 1,000,000 10, # packets at 10Gb/s Sawtooth Peak-to-trough Smoothing of many sawtooths Non-bursty arrivals Intuition & Proofs Simulated Single TCP Flow Simulations, Experiments Simulated Many TCP Flows Evidence

October 2005 Routers with Small Buffers 3 Universally applied rule-of-thumb:  A router needs a buffer size: 2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link Context  Mandated in backbone and edge routers.  Appears in RFPs and IETF architectural guidelines.  Usually referenced to Villamizar and Song: “High Performance TCP in ANSNET”, CCR,  Already known by inventors of TCP [Van Jacobson, 1988]  Has major consequences for router design C Router Source Destination 2T Backbone Router Buffers

October 2005 Routers with Small Buffers 4 Rule for adjusting W  If an ACK is received:W ← W+1/W  If a packet is lost:W ← W/2 Single TCP Flow Only W packets may be outstanding

October 2005 Routers with Small Buffers 5 Rule for adjusting W  If an ACK is received:W ← W+1/W  If a packet is lost:W ← W/2 Single TCP Flow Only W packets may be outstanding SourceDest t Window size

October 2005 Routers with Small Buffers 6 Time evolution of a single TCP flow through a router. Buffer is < 2T*C Time Evolution of a Single TCP Flow Time evolution of a single TCP flow through a router. Buffer is 2T*C

October 2005 Routers with Small Buffers 7 Synchronized Flows Aggregate window has same dynamics Therefore buffer occupancy has same dynamics Rule-of-thumb still holds. t

October 2005 Routers with Small Buffers 8 Probability Distribution B 0 Buffer Size Many TCP Flows

October 2005 Routers with Small Buffers 9 Simulation Required Buffer Size

October 2005 Routers with Small Buffers 10 Real Network Experiments Stanford University dorm traffic Network Lab (Cisco routers) at University of Wisconsin Internet2 Operational Internet backbone

October 2005 Routers with Small Buffers 11 Internet Backbone Experiment Buffer sizes 190ms, 10ms, 5ms, 2.5 and 1ms Load balancing High link utilization Long duration (about two weeks) Drops, utilization data collected every 30 seconds Test flows

October 2005 Routers with Small Buffers 12 Packet Drops vs. Link Load Buffer size = 190ms, 10ms, 5ms MAX

October 2005 Routers with Small Buffers 13 Packet Drops vs. Link Load Buffer size = 1ms

October 2005 Routers with Small Buffers 14 Relative Link Utilization Utilization of the link with 1ms buffer / Utilization of the link with 190ms buffer

October 2005 Routers with Small Buffers 15 Relative Utilization (Cont’d)

October 2005 Routers with Small Buffers 16 Theory vs. Practice M/D/1 Theory (benign conditions) Practice Typical OC192 router linecard buffers over 1,000,000 packets Can we make traffic look “Poisson-enough” when it arrives to the routers…? Poisson B D Loss independent of link rate, RTT, number of flows, etc. 5 orders of magnitude difference! 5 orders of magnitude difference!

October 2005 Routers with Small Buffers 17 Assume:  Buffer size >  Distance between consecutive packets of a single flow S > Limited injection rate  Flows are not synchronized; and  Start times picked randomly and independently We can prove that the packet drop probability is very low. Paced Injections Similar results from Cambridge/UCL, UMass and Stanford See papers in: ACM Computer Communications Review, July 2005 Similar results from Cambridge/UCL, UMass and Stanford See papers in: ACM Computer Communications Review, July 2005

October 2005 Routers with Small Buffers 18 O(log W) Buffers Assumptions:  Internet core is over-provisioned Example: Load < 80%  There is spacing between packets of the same flow: Natural: Slow access links Artificial: Paced TCP Result: Traffic is very smooth, and loss rate is very low, independent of RTT, and number of flows. With a buffer size of just packets we can gain high throughput.

October 2005 Routers with Small Buffers 19 Leaky Bucket – Paced vs. Reno Bucket drains with a constant rate. Load is 90% for both cases.

October 2005 Routers with Small Buffers 20 TCP Reno TCP Reno sends packets in a burst  High drop rate

October 2005 Routers with Small Buffers 21 Paced TCP Spacing packets  Much lower drop rate

October 2005 Routers with Small Buffers 22 O(log W) Buffers Regular TCP TCP With Pacing TCP With Pacing

October 2005 Routers with Small Buffers 23 O(log W) Buffers

October 2005 Routers with Small Buffers 24 Ideal Experiment Highly loaded link, with real/realistic traffic Precisely controlled router buffers Packet traces with precise timestamps Work in progress: Sprint, Verizion, Telcordia, Lucent, … Packet Trace Monitor Packet Trace Monitor

October 2005 Routers with Small Buffers 25 Conclusion and Future Work Theory:  Reducing buffer sizes by a factor of sqrt(N) does not affect the network performance.  Reducing the buffer sizes to O(logW) does not affect the network performance if: The network is over provisioned; and We use Paced TCP; or Have slow access links Experimental Validation:  Thousands of ns2 simulations  Stanford dorm, University of Wisconsin Testbed, Internet2, Level 3 Communications, …  Ongoing work and need your help

October 2005 Routers with Small Buffers 26 Thanks! More Info?

October 2005 Routers with Small Buffers 27 O(log W) Buffers With a large ratio between core and access link bandwidth Bottleneck Bandwidth = 1Gb/s