EE384y EE384Y: Packet Switch Architectures Part II Scaling Crossbar Switches Nick McKeown Professor of Electrical Engineering and Computer Science, Stanford University

EE384y Outline Up until now, we have focused on high performance packet switches with: 1. A crossbar switching fabric, 2. Input queues (and possibly output queues as well), 3. Virtual output queues, and 4. Centralized arbitration/scheduling algorithm. Today we’ll talk about the implementation of the crossbar switch fabric itself. How are they built, how do they scale, and what limits their capacity?

EE384y Crossbar switch Limiting factors 1. N 2 crosspoints per chip, or N x N -to-1 multiplexors 2. It’s not obvious how to build a crossbar from multiple chips, 3. Capacity of “I/O”s per chip.  State of the art: About 300 pins each operating at 3.125Gb/s ~= 1Tb/s per chip.  About 1/3 to 1/2 of this capacity available in practice because of overhead and speedup.  Crossbar chips today are limited by “I/O” capacity.

EE384y Scaling number of outputs: Trying to build a crossbar from multiple chips 4 inputs 4 outputs Building Block: 16x16 crossbar switch: Eight inputs and eight outputs required!

EE384y Scaling line-rate: Bit-sliced parallelism Linecard Cell Scheduler Cell is “striped” across multiple identical planes. Crossbar switched “bus”. Scheduler makes same decision for all slices k

EE384y Scaling line-rate: Time-sliced parallelism Linecard Scheduler Cell carried by one plane; takes k cell times. Scheduler is unchanged. Scheduler makes decision for each slice in turn k Cell

EE384y Scaling a crossbar  Conclusion: scaling the capacity is relatively straightforward (although the chip count and power may become a problem).  What if we want to increase the number of ports?  Can we build a crossbar-equivalent from multiple stages of smaller crossbars?  If so, what properties should it have?

EE384y stage Clos Network n x kn x k m x mm x m k x nk x n 1 N N = n x m k >= n 1 2 … m 1 2 … … … k 1 2 … m 1 N nn

EE384y With k = n, is a Clos network non- blocking like a crossbar? Consider the example: scheduler chooses to match (1,1), (2,4), (3,3), (4,2)

EE384y With k = n is a Clos network non- blocking like a crossbar? Consider the example: scheduler chooses to match (1,1), (2,2), (4,4), (5,3), … By rearranging matches, the connections could be added. Q: Is this Clos network “rearrangeably non-blocking”?

EE384y With k = n a Clos network is rearrangeably non-blocking Routing matches is equivalent to edge-coloring in a bipartite multigraph. Colors correspond to middle-stage switches. (1,1), (2,4), (3,3), (4,2) Each vertex corresponds to an n x k or k x n switch. No two edges at a vertex may be colored the same. Vizing ‘64: a D -degree bipartite graph can be colored in D colors. Therefore, if k = n, a 3-stage Clos network is rearrangeably non-blocking (and can therefore perform any permutation).

EE384y How complex is the rearrangement?  Method 1: Find a maximum size bipartite matching for each of D colors in turn, O( DN 2. 5 ).  Method 2: Partition graph into Euler sets, O( N.logD ) [Cole et al. ‘00]

EE384y Edge-Coloring using Euler sets  Make the graph regular: Modify the graph so that every vertex has the same degree, D. [combine vertices and add edges; O ( E )].  For D=2 i, perform i “Euler splits” and 1-color each resulting graph. This is logD operations, each of O(E).

EE384y Euler partition of a graph Euler partiton of graph G : 1.Each odd degree vertex is at the end of one open path. 2.Each even degree vertex is at the end of no open path.

EE384y Euler split of a graph Euler split of G into G 1 and G 2 : 1.Scan each path in an Euler partition. 2.Place each alternate edge into G 1 and G 2 G G1G1 G2G2

EE384y Edge-Coloring using Euler sets  Make the graph regular: Modify the graph so that every vertex has the same degree, D. [combine vertices and add edges; O ( E )].  For D=2 i, perform i “Euler splits” and 1-color each resulting graph. This is logD operations, each of O(E).

EE384y Implementation Scheduler Route connections Route connections Request graph PermutationPaths

EE384y Implementation Pros  A rearrangeably non-blocking switch can perform any permutation  A cell switch is time-slotted, so all connections are rearranged every time slot anyway Cons  Rearrangement algorithms are complex (in addition to the scheduler) Can we eliminate the need to rearrange?

EE384y Strictly non-blocking Clos Network Clos’ Theorem: If k >= 2n – 1, then a new connection can always be added without rearrangement.

EE384y I1I1 I2I2 … ImIm O1O1 O2O2 … OmOm M1M1 M2M2 … … … MkMk n x k m x m k x n 1 N N = n x m k >= n 1 N nn

EE384y Clos Theorem IaIa ObOb x x + n 1 n k 1 n k 1.Consider adding the n -th connection between 1 st stage I a and 3 rd stage O b. 2.We need to ensure that there is always some center-stage M available. 3.If k > (n – 1) + (n – 1), then there is always an M available. i.e. we need k >= 2n – 1. n – 1 already in use at input and output.

EE384y Scaling Crossbars: Summary  Scaling capacity through parallelism (bit- slicing and time-slicing) is straightforward.  Scaling number of ports is harder…  Clos network:  Rearrangeably non-blocking with k = n, but routing is complicated,  Strictly non-blocking with k >= 2n – 1, so routing is simple. But requires more bisection bandwidth.