High Performance Networking with Little or No Buffers Yashar Ganjali High Performance Networking Group Stanford University Joint work with: Guido Appenzeller, Ashish Goel, Tim Roughgarden, Nick McKeown May 5, 2005

2 Motivation Networks with Little or No Buffers  Problem  Internet traffic is doubled every year  Disparity between traffic and router growth (space, power, cost)  Possible Solution  All-Optical Networking  Consequences  Large capacity  large traffic  Little or no buffers

May 5, Which would you choose? DSL Router 1DSL Router 2 $50 4 x 10/100 Ethernet 1.5Mb/s DSL connection 1Mbit of packet buffer $55 4 x 10/100 Ethernet 1.5Mb/s DSL connection 4Mbit of packet buffer Bigger buffers are better

May 5, What we learn in school  Packet switching is good  Long haul links are expensive  Statistical multiplexing allows efficient sharing of long haul links  Packet switching requires buffers  Packet loss is bad  Use big buffers  Luckily, big buffers are cheap

May 5, Statistical Multiplexing Observations 1.The bigger the buffer, the lower the packet loss. 2.If the buffer never goes empty, the outgoing line is busy 100% of the time.

May 5, What we learn in school Queueing Theory 1 1  M/M/1 X Buffer size Loss rate Observations 1.Can pick buffer size for a given loss rate. 2.Loss rate falls fast with increasing buffer size. 3.Bigger is better.

May 5, What we learn in school  Moore’s Law: Memory is plentiful and halves in price every 18 months.  1Gbit memory holds 500k packets and costs $25.  Conclusion:  Make buffers big.  Choose the $55 DSL router.

May 5, Why bigger isn’t better  Network users don’t like buffers  Network operators don’t like buffers  Router architects don’t like buffers  We don’t need big buffers  We’d often be better off with smaller ones

May 5, Backbone Router Buffers  Universally applied rule-of-thumb:  A router needs a buffer size: 2T is the two-way propagation delay C is capacity of bottleneck line  Context  Mandated in backbone and edge routers.  Appears in RFPs and IETF architectural guidelines..  Usually referenced to Villamizar and Song: “High Performance TCP in ANSNET”, CCR,  Already known by inventors of TCP [Van Jacobson, 1988]  Has major consequences for router design C Router Source Destination 2T

May 5, Review: TCP Congestion Control Only W packets may be outstanding Rule for adjusting W  If an ACK is received: W ← W+1/W  If a packet is lost:W ← W/2

May 5, Review: TCP Congestion Control Only W packets may be outstanding Rule for adjusting W  If an ACK is received: W ← W+1/W  If a packet is lost:W ← W/2 SourceDest t Window size

May 5, Buffer Size in the Core Probability Distribution B 0 Buffer Size

May 5, Backbone router buffers  It turns out that  The rule of thumb is wrong for a core routers today  Required buffer is instead of

May 5, Simulation Required Buffer Size

May 5, Validation Theoretical results validated by:  Thousands of ns2 simulations  Network lab (Cisco routers) at University of Wisconsin  Stanford University dorm traffic  Internet2 experiments Ongoing work with network operators and router vendors…

May 5, Impact on Router Design  10Gb/s linecard with 200,000 x 56kb/s flows  Rule-of-thumb: Buffer = 2.5Gbits Requires external, slow DRAM  Becomes: Buffer = 6Mbits Can use on-chip, fast SRAM Completion time halved for short-flows  40Gb/s linecard with 40,000 x 1Mb/s flows  Rule-of-thumb: Buffer = 10Gbits  Becomes: Buffer = 50Mbits

May 5, How small can buffers be?  Imagine you want to build an all-optical router for a backbone network…  …and you can build a few dozen packets in delay lines.  Conventional wisdom: It’s a routing problem (hence deflection routing, burst- switching, etc.)  Our belief: First, think about congestion control.

May 5, TCP with ALMOST No Buffers Utilization of bottleneck link = 75%

May 5, Two Concurrent TCP Flows

May 5, TCP Throughput with Small Buffers

May 5, TCP Reno Performance

May 5, The chasm between theory and practice  M/M/1 1 1 X  = 50%, EX = 1 packet  = 75%, EX = 3 packets  = 50%, P[X>10] <  = 75%, P[X>10] < 0.06 Theory (benign conditions) Practice Typical OC192 router linecard buffers over 2,000,000 packets Can we make the traffic arriving at the routers Poisson “enough” to get most of the benefit?

May 5, Ideal Solution  If packets are spaced out perfectly; and  The starting times of flows are chosen randomly;  We only need a small buffer for contention resolution.

May 5, Pacing  We need to break bursts  Modify TCP: Instead of sending packets when your receive ACKS send packets with a fixed rate of CWND/RTT.  Rely on network properties: Access links throttle the flows to low rate Core:Acess > 1000:1 TCP’s window size is limited today.  If these properties make the flow look like poisson with only 5-10 packets of buffering we can get 70-80% throughput.

May 5, What we know so far about very small buffers Arbitrary Injection Process If Poisson Process with load < 1 Complete Centralized Control Any rate > 0 need unbounded buffers TheoryExperiment Need buffer size of approx: O(logD + logW) i.e pkts D=#of hops W=window size TCP Pacing: Results as good or better than for Poisson Constant fraction throughput with constant buffers [Leighton 1999]

May 5, CWND: Reno vs. Paced TCP

May 5, TCP Reno: Throughput vs. Buffer Size

May 5, Paced TCP: Throughput vs. Buffer Size

May 5, Early results Congested core router with 10 packet buffers. Average offered load = 80% RTT = 100ms; each flow limited to 2.5Mb/s router source server source 10Gb/s >10Gb/s

May 5, Slow access links, lots of flows, 10 packet buffers router source server source 10Gb/s 5Mb/s Congested core router with 10 packet buffers. RTT = 100ms; each flow limited to 2.5Mb/s

May 5, Conclusion  We can reduce 1,000,000 packet buffers to 10,000 today.  We can probably reduce to packet buffers:  With many small flows, no change needed  With some large flows, need pacing in the access routers or at the edge devices.  Need more experiments.

May 5, Experiments  Performance measurement with  Small (thousands of packets); and  Tiny (tens of packets) buffers  Metrics:  Link utilization (goodput/throughput)  Drops  Buffer occupancy  Etc.  Data  Gathered for minutes to days  High load (50-70% utilization) is better

Thank you! Questions?