1. High Performance All-Optical Networks with Small Buffers Yashar Ganjali High Performance Networking Group Stanford University yganjali@stanford.edu http://www.stanford.edu/~yganjali Joint work with: Prof. Ashish Goel Prof. Nick McKeown Prof. Tim Roughgarden October 4, 2004 - UCSB

2. 2 Outline  Issues specific to All-optical networks  Why do we need buffers?  Contention resolution  Congestion control  How large are buffers today?  How small can buffers be?  Congestion  Contention Scheduling Load balancing in time and space  Summary and conclusions

3. October 4, 2004 - UCSB 3 All-Optical Network Design Issues  Wavelength conversion  Without full wavelength conversion Routing problem is very hard  With full wavelength conversion Routing is the same as electronic networks  Buffering  Existing systems Huge buffers are assumed to be available  Need to reduce the buffer size

4. October 4, 2004 - UCSB 4 Leverages  Full wavelength conversion  We don’t need to worry about it  Huge amount of capacity  We can tradeoff capacity to address small buffers problem  Deflection routing  Not necessary at this stage

5. October 4, 2004 - UCSB 5  Issues specific to All-optical networks  Why do we need buffers?  Contention resolution  Congestion control  How large are buffers today?  How small can buffers be?  Congestion  Contention Scheduling Load balancing in time and space  Summary and conclusions Outline

6. October 4, 2004 - UCSB 6 Why do we need buffers?  Internet traffic is bursty in nature  Variations in Starting time Flow length Rate  Short-term fluctuations  Contention  Long-term fluctuations  Congestion

7. October 4, 2004 - UCSB 7 Contention  Contention is caused by  Synchronization of flows  Stochastic collisions

8. October 4, 2004 - UCSB 8 Congestion Control Rule for adjusting W  If an ACK is received:W ← W+1/W  If a packet is lost:W ← W/2 SourceDest t Window size Only W packets may be outstanding

9. October 4, 2004 - UCSB 9 Rule for adjusting W  If an ACK is received:W ← W+1/W  If a packet is lost:W ← W/2 Congestion Control (Cont’d) Only W packets may be outstanding

10. October 4, 2004 - UCSB 10  Issues specific to All-optical networks  Why do we need buffers?  Contention resolution  Congestion control  How large are buffers today?  How small can buffers be?  Congestion  Contention Scheduling Load balancing in time and space  Summary and conclusions Outline

11. October 4, 2004 - UCSB 11  Universally applied rule-of-thumb:  A router needs a buffer size: 2T is the two-way propagation delay C is capacity of bottleneck link  Context  Mandated in backbone and edge routers.  Appears in RFPs and IETF architectural guidelines.  Usually referenced to Villamizar and Song: “High Performance TCP in ANSNET”, CCR, 1994.  Already known by inventors of TCP [Van Jacobson, 1988] Backbone Router Buffers C Router Source Destination 2T

12. October 4, 2004 - UCSB 12 Single TCP Flow Router with large enough buffers for full link utilization

13. October 4, 2004 - UCSB 13 Single TCP Flow Router with large enough buffers for full link utilization Observation  If buffer doesn’t go empty when window size halves, then we have 100% throughput. t Window size RTT It follows that

14. October 4, 2004 - UCSB 14 Buffer = Rule of Thumb

15. October 4, 2004 - UCSB 15 Under-buffered Link

16. October 4, 2004 - UCSB 16  Issues specific to All-optical networks  Why do we need buffers?  Contention resolution  Congestion control  How large are buffers today?  How small can buffers be?  Congestion  Contention Scheduling Load balancing in time and space  Summary and conclusions Outline

17. October 4, 2004 - UCSB 17 Rule-of-thumb  Rule-of-thumb makes sense for one flow  Typical backbone link has > 20,000 flows  Does the rule-of-thumb still hold?  Answer:  If flows are perfectly synchronized, then Yes.  If flows are desynchronized then No.

18. October 4, 2004 - UCSB 18 If flows are synchronized  Aggregate window has same dynamics  Therefore buffer occupancy has same dynamics  Rule-of-thumb still holds. t

19. October 4, 2004 - UCSB 19 If flows are not synchronized Probability Distribution B 0 Buffer Size

20. October 4, 2004 - UCSB 20  It turns out that  The rule of thumb is wrong for core routers today  Required buffer is instead of  [Appenzeller, Keslassy, McKeown 2004] Backbone Router Buffers

21. October 4, 2004 - UCSB 21 Simulation Required Buffer Size

22. October 4, 2004 - UCSB 22 TCP with ALMOST no buffers Utilization of bottleneck link = 75%

23. October 4, 2004 - UCSB 23 TCP with almost no buffers  With almost no buffering and just a single flow we loose about 25% of the capacity.  Capacity is not a bottleneck anymore  More flows = less capacity loss  Huge number of flows in the core

24. October 4, 2004 - UCSB 24  Issues specific to All-optical networks  Why do we need buffers?  Contention resolution  Congestion control  How large are buffers today?  How small can buffers be?  Congestion  Contention Scheduling Load balancing in time and space  Summary and conclusions Outline

25. October 4, 2004 - UCSB 25 Contention resolution  Two extreme approaches  No scheduling: Deal with contention by having large buffers.  Tight Scheduling: Precisely schedule exact packet injection times, so that contention is minimized. No schedulingTight scheduling

26. October 4, 2004 - UCSB 26  Constraints  No buffer: If S 2 sends a packet at time t, S 1 cannot send a packet at time t+1  Buffer size B: S 1 and S 2 cannot send more than T+B packets in any interval of length T Scheduling S1S1 S2S2 D

27. October 4, 2004 - UCSB 27 Scheduling Problem  The optimal solution is hard to find in general  In no buffers case: Exact solution: NP-Complete (Job-shop scheduling) 50% Throughput: Easy to solve  With buffers: Open problem  Extreme synchronization needed  Distributed algorithm needed

28. October 4, 2004 - UCSB 28 Randomization  Randomization: Randomly delay the injection time of the packets  Alleviates short-term contention  Simple to implement  Guaranteed performance No schedulingTight scheduling ??? Randomization

29. October 4, 2004 - UCSB 29 Randomization (cont’d) Time Input #1 Input #2 Before randomization Time Input #1 Input #2 After randomization

30. October 4, 2004 - UCSB 30 Randomization (cont’d)  Shape the traffic at injection time (make Poisson)  Reshape at each router  When the load is low we can bound the loss rate Buffer size Loss rate  1 1 M/M/1 X

31. October 4, 2004 - UCSB 31 Preliminary Results Theorem. We can achieve constant factor throughput (roughly 70-80%) with very small amount of loss using buffers of size O(log L), where L is the length of the longest path in the network.  Assumption  No reactive flow control  Currently maximum window size is 12 and 64 for Linux and Windows XP

32. October 4, 2004 - UCSB 32  Issues specific to All-optical networks  Why do we need buffers?  Contention resolution  Congestion control  How large are buffers today?  How small can buffers be?  Congestion  Contention Scheduling Load balancing in time and space  Summary and conclusions Outline

33. October 4, 2004 - UCSB 33 Preliminary Results Conjecture. Routers in the core of the current Internet only need buffers of size O(log L) instead of the 2TxC.  Need to study  The interaction between randomization and congestion control  Impact of co-existing flows  Emerging applications (non-TCP or modified TCP) which will allow much larger windows per flow

34. October 4, 2004 - UCSB 34 Summary and conclusions  We need buffers to address  Contention  Congestion  Aggregation takes care of congestion  Use randomization to reduce contention  At the price of loosing some capacity, a network with small buffers can have high performance. Many thanks to Guido Appenzeller at Stanford for providing the flash animations.