Topologies

Overview Direct Networks Indirect Networks Cost Model Comparison of Direct and Indirect Networks

Classification Shared medium networks Direct networks Example: backplane buses Direct networks Example: k-ary n-cubes, meshes, and trees Indirect networks Example: multistage interconnection networks Hybrid Networks Example: hypergraph topologies

Shared Medium Networks I/O Node Buses are the most common, lowest cost option Useful for broadcast traffic Coherence management in SMPs Arbitration, width, and split transactions Centralized vs. distributed control

Full Connectivity: Crossbar 7 6 5 4 3 2 1 7 6 5 4 3 2 1 buffer Full point to point connectivity Dedicated path from each input to each output O(nm) cost for n inputs and m outputs Non-blocking but suffers from head of line blocking Buffered cross bars

Buffered Cross Bars and Switching 7 6 5 4 3 2 1 Slice 0 Byte 0 Byte 1 Slice 1 Byte (s-1) Slice (s-1) Output 0 Multiple crossbars operating in parallel across a packet to increase throughput Each slice of the packet applied to the same port of each cross bar Buffer sizing based on flow control delays

Operational Characteristics 7 6 5 4 3 2 1 Input queuing (IQ), output queuing (OQ), combined input-output queuing (CIOQ) Head of line blocking and switch speedup Speedup of S implies S packets can moved from each input to outputs in a packet interval arrival time

Direct Networks Buses do not scale, electrically or in bandwidth Processor+ $C Memory Ejection channels injection channels Router Buses do not scale, electrically or in bandwidth Full connectivity too expensive (not the same as Xbars) Network built on point-to-point transfers

System Views Node Node Node Processor NB I/O SB NI Chip boundary today From http://www.psc.edu/publications/tech_reports/PDIO/CrayXT3-ScalableArchitecture.jpg I/O Processor NB Chip boundary today PCIe High latency region SB NI

Common Topologies Definition Basic connectivity properties Diameter I/O (also referred to as node size or pin-out) Bisection bandwidth Routing (later)

Common Properties Diameter Node degree Bisection BW Channel length Regularity and symmetry Latency I/O BW (pin-out) Throughput Routing and path diversity

Multidimensional Mesh Asymmetric topology We have nodes with nodes in each dimension  the radix The local port on each router to the local node is not shown Each node addressed by its coordinates

Evaluation Metrics Bisection bandwidth This is minimum bandwidth across any bisection of the network Bisection bandwidth is a limiting attribute of performance

n-dimensional Mesh When all dimensions have the same radix and we have nodes Network Diameter (bidirectional channels) Must cross at most links in every dimension Maximum node Degree is 2Wn (Wn at the corners) Bisection Bandwidth is Wkn-1

n-dimensional Mesh Network Diameter (directional channels) Must cross at most links in each dimension i Maximum node Degree is 2Wn (Wn at the corners) Bisection width is where is the largest radix Take the cut orthogonal to the largest dimension

Only one wrap around link is shown Tori Only one wrap around link is shown 2D 3D Binary hypercube

Multidimensional Tori A symmetric generalization of the mesh We have nodes with nodes in each dimension  the radix Two nodes X and Y are adjacent if and only if except for dimension j where When n=1 we have the ring When we have nodes for a k-ary n-cube

k-ary n-cubes When all dimensions have the same radix and we have nodes Network Diameter (bidirectional channels) Must cross at most links in every dimension Node Degree is Bisection Bandwidth is 2Wkn-1

Multidimensional Tori Network Diameter (directional channels) Must cross at most links in each dimension i Node Degree is Bisection width is where is the largest radix Take the cut orthogonal to the largest dimension

Routing in Multidimensional Networks Basic routing algorithms are dimension order This ensures deadlock and livelock freedom Choice of dimension order is irrelevant to deadlock and livelock freedom as long as it is fixed! Routing algorithms are shortest path Offsets in each dimension are computed from the corresponding digits of the addresses

Binary Hypercube With and nodes we have the binary hypercube Node degree is Network diameter is Addressing, connectivity, and Hamming distance

Routing in Binary Hypercubes Follows from the preceding but has a few special properties Path computation Take the exclusive-OR of the source destination order Traverse all dimensions where source and destination differ Number of hops (path length) is the hamming distance between the source and destination addresses. Route header is now just a bit vector and routing is a prefix computation

Routing in Binary Hypercubes 1100 1101 1000 0000 The routing vector is successively corrected until it is 0. Hardware implementation is simple and fast Also known as the e-cube routing algorithm

Comparison of Metrics Network Bisection Width Node Size k-ary n-cube 2Wkn-1 2Wn Binary n-cube NW/2 nW n-dimensional mesh Wkn-1 These topologies are also referred to as (strongly orthogonal) topologies

Generalized Hypercubes Generalization of tori in multiple dimensions and multiple radices Full connectivity in each dimension rather than just to nearest neighbhors Preserves the structure of addressing and routing techniques or orthogonal topologies

Less Common Topologies nodes Size of ring is n Constant node degree Compact diameter but long average path length Connectivity Connectivity is defined by the permutations of digits

Less Common Topologies (cont.) Routing structure Basic properties A note on irregular direct topologies

Engineering Considerations Average channel wire length Distinguish between layout (physical) and topology (logical)

Extensions to Higher Dimensions Interleaved layout significant reduces the wire/cable length Improves packaging modularity Note the end-around connections Impacts performance and cost Adapted from “Scalable Switching Fabrics for Internet Routers,” by W. J. Dally (can be found at www.avici.com)

Indirect Networks Switches may or may not host end-points

Multistage Interconnection Networks Interconnect specified as a permutation 7 6 5 4 3 2 1 7 6 5 4 3 2 1 Number of stages = log2N Can be generalized to KxK switches Networks defined by inter-stage permutations Switch states © T.M. Pinkston, J. Duato, with major contributions by J. Filch

The Shuffle Interconnection

The Baseline Interconnection

The Butterfly Interconnection

The Cube Interconnection switch cube(i)

© T.M. Pinkston, J. Duato, with major contributions by J. Filch Omega Network 0000 0000 0001 0001 0010 0010 0011 0011 0100 0100 0101 0101 0110 0110 0111 0111 1000 1000 1001 1001 1010 1010 1011 1011 1100 1100 1101 1101 1110 1110 1111 1111 shuffle © T.M. Pinkston, J. Duato, with major contributions by J. Filch

Omega Network: Functional View

© T.M. Pinkston, J. Duato, with major contributions by J. Filch Baseline Network 0000 0000 0001 0001 0010 0010 0011 0011 0100 0100 0101 0101 0110 0110 0111 0111 1000 1000 1001 1001 1010 1010 1011 1011 1100 1100 1101 1101 1110 1110 1111 1111 1 2 4 sub-shufflen-i © T.M. Pinkston, J. Duato, with major contributions by J. Filch

© T.M. Pinkston, J. Duato, with major contributions by J. Filch Butterfly 0000 0000 0001 0001 0010 0010 2-ary 4-fly 0011 0011 0100 0100 0101 0101 0110 0110 0111 0111 1000 1000 1001 1001 1010 1010 1011 1011 1100 1100 1101 1101 1110 1110 1111 1111 4 1 butterflyi dimensions © T.M. Pinkston, J. Duato, with major contributions by J. Filch

© T.M. Pinkston, J. Duato, with major contributions by J. Filch Cube Network 0000 0000 0001 0001 0010 0010 0011 0011 0100 0100 0101 0101 0110 0110 0111 0111 1000 1000 1001 1001 1010 1010 1011 1011 1100 1100 1101 1101 1110 1110 1111 1111 1 4 butterflyn-2 © T.M. Pinkston, J. Duato, with major contributions by J. Filch

Routing in MINs Routing can be modeled as a sequence address transformations Each stage transforms a bit of the source address into a bit of the destination address Routing Implementation: a single bit of the destination address determines the output port Examples

© T.M. Pinkston, J. Duato, with major contributions by J. Filch Omega Network 0000 0000 0001 0001 0010 0010 0011 0011 0100 0100 0101 0101 0110 0110 0111 0111 1000 1000 1001 1001 1010 1010 1011 1011 1100 1100 1101 1101 1110 1110 1111 1111 shuffle © T.M. Pinkston, J. Duato, with major contributions by J. Filch

Basic Properties Diameter, path length and pin-out Bisection bandwidth Number of unique paths

Blocking vs. Non-blocking Networks 7 6 5 4 3 2 1 7 6 5 4 3 2 1 X 7 6 5 4 3 2 1 non-blocking topology blocking topology Consider the permutation behavior Model the input-output requests as permutations of the source addresses © T.M. Pinkston, J. Duato, with major contributions by J. Filch

Blocking Behavior Strictly non-blocking Weakly non-blocking A new connection can always be set up Every permutation can be realized Weakly non-blocking Strictly non-blocking only under some routing protocols Rearrangeable Every permutation can be realized by rearranging existing connections Blocking Some permutations cannot be realized

© T.M. Pinkston, J. Duato, with major contributions by J. Filch Crossbar Network 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 11 11 12 12 13 13 14 14 15 15 © T.M. Pinkston, J. Duato, with major contributions by J. Filch

Non-Blocking Clos Network 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 11 11 12 12 13 13 14 14 15 15 © T.M. Pinkston, J. Duato, with major contributions by J. Filch

Blocking Behavior Properties depend on relative values of n and m Rearrangeable when Non-blocking when

Clos Network Properties General 3 stage non-blocking network Originally conceived for telephone networks Recursive decomposition Produces the Benes network with 2x2 switches

Clos Network: Recursive Decomposition 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 11 11 12 12 13 13 14 14 15 15 16 port, 5-stage Clos network © T.M. Pinkston, J. Duato, with major contributions by J. Filch

Clos Network: Recursive Decomposition 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 11 11 12 12 13 13 14 14 15 15 16 port, 7 stage Clos network = Benes topology © T.M. Pinkston, J. Duato, with major contributions by J. Filch

© T.M. Pinkston, J. Duato, with major contributions by J. Filch Path Diversity 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 11 11 12 12 13 13 14 14 15 15 Alternative paths from 0 to 1. 16 port, 7 stage Clos network = Benes topology © T.M. Pinkston, J. Duato, with major contributions by J. Filch

© T.M. Pinkston, J. Duato, with major contributions by J. Filch Path Diversity 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 11 11 12 12 13 13 14 14 15 15 Alternative paths from 4 to 0. 16 port, 7 stage Clos network = Benes topology © T.M. Pinkston, J. Duato, with major contributions by J. Filch

© T.M. Pinkston, J. Duato, with major contributions by J. Filch Path Diversity 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 11 11 12 12 13 13 14 14 15 15 Contention free, paths 0 to 1 and 4 to 1. 16 port, 7 stage Clos network = Benes topology © T.M. Pinkston, J. Duato, with major contributions by J. Filch

Bidirectional MINs

Routing in Bidirectional MINS Networks are multi-path Routing takes place in two steps: route to an intermediate node followed by routing to destination Multiple intermediate nodes can be selected Path from intermediate node to destination is unique

Generalized MINs Generalized switch radix Routing and mathematics uniform across switch radix values

Routing in MINs “Square” vs. non-square MINs vs. dilated MINs 7 6 5 4 3 2 1 7 6 5 4 3 2 1 “Square” vs. non-square MINs vs. dilated MINs Permutation routing and centralized control Message routing, distributed control, and contention Basic elements of a logkN network and topological equivalence Single source destination path Logarithmic delay Routing in MINs

The Blocking Condition sn-1,sn-2,…,s2,s1,s0,dn-1,dn-2,…,d2,d1,d0 Equal? rn-1,rn-2,…,r2,r1,r0,tn-1,tn-2,…,t2,t1,t0 Addresses of ports at intermediate switches are computed from the source-destination addresses Blocking condition: two paths collide iff they compete for the same link/port at any stage Routing in MINs

Self Routed MINs Routing in MINs

© T.M. Pinkston, J. Duato, with major contributions by J. Filch Baseline Network 0000 0000 0001 0001 0010 0010 0011 0011 0100 0100 0101 0101 0110 0110 0111 0111 1000 1000 1001 1001 1010 1010 1011 1011 1100 1100 1101 1101 1110 1110 1111 1111 1 2 4 sub-shufflen-i © T.M. Pinkston, J. Duato, with major contributions by J. Filch

Self Routed MINs (cont.) Self routing property Routing decision a function of only the destination Computation of destination routing tag tn-1,tn-2,…,t2,t1,t0 ti = di+1 0 <= i <= n-2, and tn-1 = d0 for butterfly MINs ti = dn-i-1 0 <= i <= n-1 for Omega and Cube networks Routing in MINs

Self Routed MINs: example Routing in MINs

Routing in Bidirectional MINs Routing uses the function FirstDifference(S,D) Identifies the nearest stage to “turnaround” Multiple choices of switches at last stage Can randomize forward path selection Routing in MINs

© T.M. Pinkston, J. Duato, with major contributions by J. Filch Butterfly 0000 0000 0001 0001 0010 0010 2-ary 4-fly 0011 0011 0100 0100 0101 0101 0110 0110 0111 0111 1000 1000 1001 1001 1010 1010 1011 1011 1100 1100 1101 1101 1110 1110 1111 1111 butterfly1 dimensions © T.M. Pinkston, J. Duato, with major contributions by J. Filch

Research Problems Reliable communication Power optimizations Exploit path diversity Adaptive routing Power optimizations On-off links Minimizing buffers for deadlock free routing Microarchitecture Low latency routers Power efficient routers

Summary and Research Directions Use of hybrid interconnection networks Best way to utilize existing pin-out? Engineering considerations rapidly prune the space of candidate topologies Routing + switching + topology = network Onto routing…….