High Performance All-Optical Networks with Small Buffers Yashar Ganjali High Performance Networking Group Stanford University Joint work with: Prof. Ashish Goel Prof. Nick McKeown Prof. Tim Roughgarden October 4, UCSB

2 Outline  Issues specific to All-optical networks  Why do we need buffers?  Contention resolution  Congestion control  How large are buffers today?  How small can buffers be?  Congestion  Contention Scheduling Load balancing in time and space  Summary and conclusions

October 4, UCSB 3 All-Optical Network Design Issues  Wavelength conversion  Without full wavelength conversion Routing problem is very hard  With full wavelength conversion Routing is the same as electronic networks  Buffering  Existing systems Huge buffers are assumed to be available  Need to reduce the buffer size

October 4, UCSB 4 Leverages  Full wavelength conversion  We don’t need to worry about it  Huge amount of capacity  We can tradeoff capacity to address small buffers problem  Deflection routing  Not necessary at this stage

October 4, UCSB 5  Issues specific to All-optical networks  Why do we need buffers?  Contention resolution  Congestion control  How large are buffers today?  How small can buffers be?  Congestion  Contention Scheduling Load balancing in time and space  Summary and conclusions Outline

October 4, UCSB 6 Why do we need buffers?  Internet traffic is bursty in nature  Variations in Starting time Flow length Rate  Short-term fluctuations  Contention  Long-term fluctuations  Congestion

October 4, UCSB 7 Contention  Contention is caused by  Synchronization of flows  Stochastic collisions

October 4, UCSB 8 Congestion Control Rule for adjusting W  If an ACK is received:W ← W+1/W  If a packet is lost:W ← W/2 SourceDest t Window size Only W packets may be outstanding

October 4, UCSB 9 Rule for adjusting W  If an ACK is received:W ← W+1/W  If a packet is lost:W ← W/2 Congestion Control (Cont’d) Only W packets may be outstanding

October 4, UCSB 10  Issues specific to All-optical networks  Why do we need buffers?  Contention resolution  Congestion control  How large are buffers today?  How small can buffers be?  Congestion  Contention Scheduling Load balancing in time and space  Summary and conclusions Outline

October 4, UCSB 11  Universally applied rule-of-thumb:  A router needs a buffer size: 2T is the two-way propagation delay C is capacity of bottleneck link  Context  Mandated in backbone and edge routers.  Appears in RFPs and IETF architectural guidelines.  Usually referenced to Villamizar and Song: “High Performance TCP in ANSNET”, CCR,  Already known by inventors of TCP [Van Jacobson, 1988] Backbone Router Buffers C Router Source Destination 2T

October 4, UCSB 12 Single TCP Flow Router with large enough buffers for full link utilization

October 4, UCSB 13 Single TCP Flow Router with large enough buffers for full link utilization Observation  If buffer doesn’t go empty when window size halves, then we have 100% throughput. t Window size RTT It follows that

October 4, UCSB 14 Buffer = Rule of Thumb

October 4, UCSB 15 Under-buffered Link

October 4, UCSB 16  Issues specific to All-optical networks  Why do we need buffers?  Contention resolution  Congestion control  How large are buffers today?  How small can buffers be?  Congestion  Contention Scheduling Load balancing in time and space  Summary and conclusions Outline

October 4, UCSB 17 Rule-of-thumb  Rule-of-thumb makes sense for one flow  Typical backbone link has > 20,000 flows  Does the rule-of-thumb still hold?  Answer:  If flows are perfectly synchronized, then Yes.  If flows are desynchronized then No.

October 4, UCSB 18 If flows are synchronized  Aggregate window has same dynamics  Therefore buffer occupancy has same dynamics  Rule-of-thumb still holds. t

October 4, UCSB 19 If flows are not synchronized Probability Distribution B 0 Buffer Size

October 4, UCSB 20  It turns out that  The rule of thumb is wrong for core routers today  Required buffer is instead of  [Appenzeller, Keslassy, McKeown 2004] Backbone Router Buffers

October 4, UCSB 21 Simulation Required Buffer Size

October 4, UCSB 22 TCP with ALMOST no buffers Utilization of bottleneck link = 75%

October 4, UCSB 23 TCP with almost no buffers  With almost no buffering and just a single flow we loose about 25% of the capacity.  Capacity is not a bottleneck anymore  More flows = less capacity loss  Huge number of flows in the core

October 4, UCSB 24  Issues specific to All-optical networks  Why do we need buffers?  Contention resolution  Congestion control  How large are buffers today?  How small can buffers be?  Congestion  Contention Scheduling Load balancing in time and space  Summary and conclusions Outline

October 4, UCSB 25 Contention resolution  Two extreme approaches  No scheduling: Deal with contention by having large buffers.  Tight Scheduling: Precisely schedule exact packet injection times, so that contention is minimized. No schedulingTight scheduling

October 4, UCSB 26  Constraints  No buffer: If S 2 sends a packet at time t, S 1 cannot send a packet at time t+1  Buffer size B: S 1 and S 2 cannot send more than T+B packets in any interval of length T Scheduling S1S1 S2S2 D

October 4, UCSB 27 Scheduling Problem  The optimal solution is hard to find in general  In no buffers case: Exact solution: NP-Complete (Job-shop scheduling) 50% Throughput: Easy to solve  With buffers: Open problem  Extreme synchronization needed  Distributed algorithm needed

October 4, UCSB 28 Randomization  Randomization: Randomly delay the injection time of the packets  Alleviates short-term contention  Simple to implement  Guaranteed performance No schedulingTight scheduling ??? Randomization

October 4, UCSB 29 Randomization (cont’d) Time Input #1 Input #2 Before randomization Time Input #1 Input #2 After randomization

October 4, UCSB 30 Randomization (cont’d)  Shape the traffic at injection time (make Poisson)  Reshape at each router  When the load is low we can bound the loss rate Buffer size Loss rate  1 1 M/M/1 X

October 4, UCSB 31 Preliminary Results Theorem. We can achieve constant factor throughput (roughly 70-80%) with very small amount of loss using buffers of size O(log L), where L is the length of the longest path in the network.  Assumption  No reactive flow control  Currently maximum window size is 12 and 64 for Linux and Windows XP

October 4, UCSB 32  Issues specific to All-optical networks  Why do we need buffers?  Contention resolution  Congestion control  How large are buffers today?  How small can buffers be?  Congestion  Contention Scheduling Load balancing in time and space  Summary and conclusions Outline

October 4, UCSB 33 Preliminary Results Conjecture. Routers in the core of the current Internet only need buffers of size O(log L) instead of the 2TxC.  Need to study  The interaction between randomization and congestion control  Impact of co-existing flows  Emerging applications (non-TCP or modified TCP) which will allow much larger windows per flow

October 4, UCSB 34 Summary and conclusions  We need buffers to address  Contention  Congestion  Aggregation takes care of congestion  Use randomization to reduce contention  At the price of loosing some capacity, a network with small buffers can have high performance. Many thanks to Guido Appenzeller at Stanford for providing the flash animations.