Isaac Keslassy, Shang-Tse (Da) Chuang, Nick McKeown Stanford University The Load-Balanced Router

2 R R R R R R Typical Router Architecture Input Switch Fabric Scheduler Output

3  Traffic matrix:  Uniform traffic matrix: λ ij = λ Definitions: Traffic Matrix R R R R R R 1 N i 1 N j

4  100% throughput: for any traffic matrix of row and column sum less than R, λ ij < μ ij Definitions: 100% Throughput R R R R R R 1 N i 1 N j

5 Router Wish List Scale to High Linecard Speeds  No Centralized Scheduler  Optical Switch Fabric  Low Packet-Processing Complexity Scale to High Number of Linecards  High Number of Linecards  Arbitrary Arrangement of Linecards Provide Performance Guarantees  100% Throughput Guarantee  Delay Guarantee  No Packet Reordering

6 Stanford 100Tb/s Router  “Optics in Routers” project   Some challenging numbers:  100Tb/s  160Gb/s linecards  640 linecards

7 In Out R R R R R R Router capacity = NR Switch capacity = N 2 R 100% Throughput in a Mesh Fabric ? ? ? ? ? ? ? ? ? R R R R R R R R R R R R R

8 R In Out R R R R R R/N If Traffic Is Uniform R R

9 Real Traffic is Not Uniform R In Out R R R R R R/N R R R R R R R R R ?

10 Out R R R R/N Load-Balanced Switch Load-balancing stageForwarding stage In Out R R R R/N R R R 100% throughput for weakly mixing traffic (Valiant, C.-S. Chang et al.)

11 Out R R R R/N In R R R R/N Load-Balanced Switch

12 Out R R R R/N In R R R R/N Load-Balanced Switch

13 Out R R R R/N In R R R R/N Intuition: Proof of 100% Throughput  Arrivals to second mesh:  Capacity of second mesh:  Second mesh: arrival rate < service rate

14 Alternative: Crossbar Switch Fabric External Outputs Intermediate ports 1 N External Inputs 1 N 1 N  Proposed by C.-S.Chang et al.  Essential result: same rate => same guarantees

15 Router Wish List Scale to High Linecard Speeds  No Centralized Scheduler  Optical Switch Fabric  Low Packet-Processing Complexity Scale to High Number of Linecards  High Number of Linecards  Arbitrary Arrangement of Linecards Provide Performance Guarantees  100% Throughput Guarantee  Delay Guarantee  No Packet Reordering ?

16 Out R R R R/N In R R R R/N Packet Reordering 1 2

17 Out R R R R/N In R R R R/N Bounding Delay Difference Between Middle Ports 1 2

18 Out R R R R/N In R R R R/N UFS (Uniform Frame Spreading) 1 2

19 Out R R R R/N In R R R R/N FOFF (Full Ordered Frames First) 1 2

20 FOFF (Full Ordered Frames First)  Input Algorithm  N FIFO queues corresponding to the N output flows  Spread each flow uniformly: if last packet was sent to middle port k, send next to k+1.  Every N time-slots, pick a flow: - If full frame exists, pick it and spread like UFS - Else if all frames are partial, pick one in round-robin order and send it N

21 Out R R R R/N In R R R R/N Bounding Reordering 1 2 3

22 FOFF  Output properties  N FIFO queues corresponding to the N middle ports  If there are N 2 packets, one of the head-of-line packets is in order and can depart   Buffer size at most N 2 packets Output 4 N

23 FOFF Properties  Property 1: FOFF maintains packet order.  Property 2: FOFF has O(1) complexity.  Property 3: Congestion buffers operate independently.  Property 4: FOFF maintains an average packet delay within constant from ideal output-queued router.  Corollary: FOFF has 100% throughput for any adversarial traffic.

24 In Out R R R R R R Output-Queued Router ? ? ? ? ? ? ? ? ? R R R R R R R R R R R R R

25 Router Wish List Scale to High Linecard Speeds  No Centralized Scheduler  Optical Switch Fabric  Low Packet-Processing Complexity Scale to High Number of Linecards  High Number of Linecards  Arbitrary Arrangement of Linecards Provide Performance Guarantees  100% Throughput Guarantee  Delay Guarantee  No Packet Reordering

26 Out R R R R/N In R R R R/N From Two Meshes to One Mesh One linecard In Out

27 From Two Meshes to One Mesh First mesh In Out In Out In Out In Out One linecard Second mesh R R R R R

28 From Two Meshes to One Mesh Combined mesh In Out In Out In Out In Out 2R R

29 Many Fabric Options Options Space: Full uniform mesh Time: Round-robin crossbar Wavelength: Static WDM Any spreading device C 1, C 2, …, C N C1C1 C2C2 C3C3 CNCN In Out In Out In Out In Out N channels each at rate 2R/N One linecard

30 AWGR (Arrayed Waveguide Grating Router) A Passive Optical Component  Wavelength i on input port j goes to output port (i+j-1) mod N  Can shuffle information from different inputs  1,  2 …  N NxN AWGR Linecard 1 Linecard 2 Linecard N  1  2  N Linecard 1 Linecard 2 Linecard N

31 In Out In Out In Out In Out Static WDM Switching: Packaging AWGR Passive and Almost Zero Power A B C D A, B, C, D A, A, A, A B, B, B, B C, C, C, C D, D, D, D N WDM channels, each at rate 2R/N

32 Router Wish List Scale to High Linecard Speeds  No Centralized Scheduler  Optical Switch Fabric  Low Packet-Processing Complexity Scale to High Number of Linecards  High Number of Linecards  Arbitrary Arrangement of Linecards Provide Performance Guarantees  100% Throughput Guarantee  Delay Guarantee  No Packet Reordering

33 Scaling Problem  For N < 64, an AWGR is a good solution.  We want N = 640.  Need to decompose.

34 A Different Representation of the Mesh In Out In Out In Out In Out R 2R Mesh 2R In Out In Out In Out In Out R 2R R

35 A Different Representation of the Mesh In Out In Out In Out In Out R In Out In Out In Out In Out R 2R/N

Example: N= R/8

37 When N is Too Large Decompose into groups (or racks) 4R/4 2R2R2R2R R2R 2R2R R

38 When N is Too Large Decompose into groups (or racks) 12L 2R 12L Group/Rack 1 Group/Rack G 12L 2R Group/Rack 1 12L 2R Group/Rack G 2RL 2RL/G Electronics Optics

39 Router Wish List Scale to High Linecard Speeds  No Centralized Scheduler  Optical Switch Fabric  Low Packet-Processing Complexity Scale to High Number of Linecards  High Number of Linecards  Arbitrary Arrangement of Linecards Provide Performance Guarantees  100% Throughput Guarantee  Delay Guarantee  No Packet Reordering

40 When Linecards are Missing 12L 2R 12L Group/Rack 1 Group/Rack G 12L 2R Group/Rack 1 12L 2R Group/Rack G 2RL 2RL/G 2RL Solution: replace mesh with sum of permutations = + + 2RL/G ≤ 2RL 2RL/G G *

41 MEMS-Based Architecture 1 2 L 1 2 L Group/Rack 1 Group/Rack G 1 2 L 1 2 L Static MEMS Switch Static MEMS Switch Electronics Optics Group/Rack 1 Group/Rack G Uniform Multiplexing Uniform Demultiplexing

42 12L12L Group/Rack 1 Group/Rack G 12L Group/Rack 1 12L Group/Rack G MEMS Switch MEMS Switch When Linecards are Missing

43 Implementation of a 100Tb/s Load-Balanced Router Linecard Rack 1 L = Gb/s linecards x 40 static MEMS Switch Rack < 100W L = Gb/s linecards Linecard Rack G = 40 L = Gb/s linecards

44 Summary  The load-balanced switch  Does not need any centralized scheduling  Can use a mesh  Using FOFF  It keeps packets in order  It guarantees 100% throughput  Using the MEMS-based architecture  It scales to high port numbers  It tolerates linecard failure

45 References  Initial Work  C.-S. Chang, D.-S. Lee and Y.-S. Jou, "Load Balanced Birkhoff- von Neumann Switches, part I: One-Stage Buffering," Computer Communications, Vol. 25, pp ,  Extensions  I. Keslassy, S.-T. Chuang, K. Yu, D. Miller, M. Horowitz, O. Solgaard and N. McKeown, "Scaling Internet Routers Using Optics," ACM SIGCOMM'03, Karlsruhe, Germany, August  I. Keslassy, S.-T. Chuang and N. McKeown, “ A Load-Balanced Switch with an Arbitrary Number of Linecards, ” IEEE Infocom ’ 04, Hong Kong, March 2004.

Thank you.