1 Maintaining Packet Order in Two-Stage Switches Isaac Keslassy, Nick McKeown Stanford University

2 Two Successive Scaling Problems OQ routers: + work-conserving - memory bandwidth = (N+1)R R R R R IQ routers: + memory bandwidth = 2R - arbitration complexity Bipartite Matching R R

3 Today: 64 ports at 10Gbps, 64-byte cells. Arbitration Time = = 51.2ns Request/Grant Communication BW = 17.5Gbps 10Gbps 64bytes IQ Arbitration Complexity Two main alternatives for scaling: 1.Increase cell size (Kar et al., 2000) 2.Eliminate arbitration (Chang et al., 2001) Scaling to 160Gbps: Arbitration Time = 3.2ns Request/Grant Communication BW = 280Gbps

4 Desirable Characteristics for Router Architecture Ideal: OQ 100% throughput Minimum delay Maintains packet order Necessary: able to regularly connect any input to any output What if the world was perfect? Assume Bernoulli iid uniform arrival traffic...

5 Cyclic Shift? 1 N 1 N Uniform Bernoulli iid traffic: 100% throughput

6 Cyclic Shift? 1 N 1 N b(t) (t) q(t) Problem: real traffic is non-uniform Long-term service opportunities exceed arrivals:

7 Two-Stage Switch External Outputs Internal Inputs 1 N External Inputs Load-balancing cyclic shift Switching cyclic shift 1 N 1 N % throughput for broad range of traffic types (C.S. Chang et al., 2001)

8 Two-Stage Switch External Outputs Internal Inputs 1 N External Inputs 1 N 1 N 1 (t) 2 (t) b(t) q(t) a(t) Traffic rate: First cyclic shift: Long-term service opportunities exceed arrivals: ( 2 = 1 possible)

9 Eliminates arbitration 100% throughput Conventional router packaging Two-Stage Switch Characteristics

Phase 2 Phase 1 Using a Single Stage Twice Lookup Buffer Lookup Buffer Lookup Buffer Linecards 1 N 1 N 1 N

11 Racks of linecards Optical links Two-Stage Switch Characteristics

12 Problem: unbounded mis-sequencing External Outputs Internal Inputs 1 N External Inputs Cyclic Shift 1 N 1 N Two-Stage Switch Characteristics

13 Full Frames First (FFF): Intuitive Idea External Outputs Internal Inputs 1 N External Inputs Cyclic Shift 1 N 1 N Idea: 1. Spread cells evenly across all linecards t 12

14 Full Frames First (FFF): Intuitive Idea Idea: 1. Spread cells evenly across all linecards 2. Read them in order External Outputs Internal Inputs 1 N External Inputs Cyclic Shift 1 N 1 N

15 Full Frames First (FFF): Intuitive Idea Idea: 1. Spread cells evenly across all linecards 2. Read them in order External Outputs Internal Inputs 1 N External Inputs Cyclic Shift 1 N 1 N

16 First Problem Problem: if two packets dont arrive consecutively, there may be a hole in the reading sequence External Outputs Internal Inputs 1 N External Inputs Cyclic Shift 1 N 1 N t 123

17 Coordination Buffer Solution: collect cells from a flow in a coordination buffer, and load-balance them among linecards 1 N Cyclic Shift 1 N 1 N Flow Load Balancing Coordination Buffer (VOQ) 1 t 123

18 Second Problem Problem: No access to cell 2 because of head-of- line blocking 1 N Cyclic Shift 1 N 1 N Flow Load Balancing Coordination Buffer (VOQ) 1 3 a b t t a b Input 1: Input 2:

19 Expanding VOQ Structure Solution: expand VOQ structure by distinguishing among switch inputs a b

20 FFF: Guarantees Theorem 2: for any arrival process, D avg (FFF) D avg (OQ) + (4N 2 - 2) Theorem 1: for any arrival process for which OQ has 100% throughput, so does FFF Theorem 3: FFF maintains packet order

21 Two-Stage Switch in Optics R/N Passive mesh R/N Passive mesh R/N Cyclic Shift RR

22 Summary FFF: practical algorithm that solves mis-sequencing Same throughput as OQ, and average delay within a bound New approach to optical switching