1. InterConnection Network Topologies to Minimize graph diameter: Low Diameter Regular graphs and Physical Wire Length Constrained networks Nilesh Choudhury Parallel Programming Lab Department of Computer Science University of Illinois, Urbana Champaign

2. Motivation ● Running a program on a large number of processors: – Large number of partitions – Large amount of communication ● Communication is the most common bottleneck for scaling a problem to large number of machines – Point to Point communication times increase ● Average hop count has increased – Collective communication times increase ● Need to send larger number of messages ● Average hop count has increased ● Diameter has increased, so max. communication time is larger – Might be limited by the maximum time

3. Solutions ● Software communication Optimizations ● Mapping your partitions (sequential entities) to minimize communication ● Interconnection Network itself could be optimized – Minimize diameter of the network to decrease global collective operations ● Broadcast ● Reduction – Amount of bandwidth used ● Minimize the total number of hops for all messages ● Minimize average number of hops

4. Scope of this talk ● Interconnection Network design Optimizations ● Routing Algorithms for these networks ● Tradeoff between – Average hop distance – Maximum hop distance (diameter) – Simplicity of routing algorithm ● Must be implemented in hardware (in as few clock cycles as possible)

5. Networks ● Direct Network – Each node is connected to a corresponding router – # routers = # nodes – Also called router-based networks ● Indirect Network – A number of nodes is connected to one switch – Fat-trees are an example

6. Network Parameters ● Degree of a node – Connectivity of the node ● Bisection bandwidth – The minimum bidirectional capacity of a network between two equally sized partitions of the network ● Diameter – Length of the longest shortest path between any two nodes in the network ● Average length of shortest path between all pairs of nodes

7. Common Networks ● HyperCube (N nodes): – Degree = logN – Diameter = logN – Bisection BW = N/2 – Avg. Internode Distance = (logN)/2 ● Fat Tree (k-ary n-tree) (N=k^n nodes): – Degree = k – Diameter = 2n – Bisection BW = N – Avg. Internode Distance = n

8. Moore Graphs ● Moore graph: – N(d,k) <= (d(d-1)^k -2) / (d-2) ● Very few graphs found that satisfy the Moore bound ● N(nodes, degree, diameter) ● Petersen graph N(10, 3, 2) ● Hoffman-Singleton graph N(50, 7, 2) ● N(3250, 57, 2) – possible but yet undiscovered

9. Low Diameter Regular Graph ● Each node has same degree logN as of a hypercube ● Diameter of LDR is 2, that for hypercube is 3 ● Average Internode distance for LDR is 1.375, while that for hypercube is 1.5

10. Diameter: LDR graph Vs Hypercube

11. How to generate a LDR graph? ● LDR graph is built based on a spanning tree ● No. of nodes = N ● Degree = k ● Start with the root. Connect it to k children ● For each of the children connect them to k-1 children (each has a parent) ● Till we have used all N nodes ● Leafs still have unconnected edges, which could be used to decrease the diameter, etc.

12. How to generate a LDR graph? contd... ● To choose the incomplete connections for leaves: – Pick a vertex 'A' with max incomplete connections – Pick another vertex 'B' randomly from remaining – If (A->B) pick another vertex – Continue the previous step till there are no vertices remaining which satisfy the condition or we find a legitimate vertex – If we find a legitimate vertex, add an edgeA->B – Else, we disconnect some edge X-Y and connect A-X and B-Y.

13. Building a LDR graph

14. Routing on a LDR graph ● Hamming for hypercube is simple (XOR op) ● Deterministic for LDR – shortest path routing ● Table driven ● Need adaptivity in the presence of network contention! ● 64 / 2048 node LDR and hypercube using deterministic and adaptive routing

15. How difficult is it to place a hypercube / LDR in physical space? ● A hypercube with N nodes – LogN dimensions ● Real world is 3D ● Difficult to place nodes in 'n' dimensions ● Really big machines (Bluegene, RedStorm) – Use 3D torus or similar – Easier to place them in physical space – Large wire lengths mean large delays, not to mention cost ● We believe for large machines, topology should consider physical placement

16. Framing the problem? ● The problem – a network with connectivity 'k'; – maximum allowable wire length is 'd' (hops); – Design a network topology within these constraints, with lowest diameter, average all pair internode distance – Also it should have a simple routing algorithm

17. Connected Graph ● In 2D: – X+, X-, Y+, Y- – These 4 connections provide connected graph ● In 3D: – X+, X-, Y+, Y-, Z+, Z- – These 6 connections provide connected graph

18. The proposed topology ● The Remaining connections are to be used to decrease the diameter and average all pair internode distance ● i=1; ● Add diagonal connections of length 'd'/i along all four directions ● i *= 2; ● Repeat the above step while total # connections are less than 'k'

19. Higher dimensions connectivity of a single node ● The picture shows 2D connectivity of a single node ● d/(sqrt(2)); d/(2sqrt(2)); d/(4sqrt(2));.... ● Similarily, for 3D, connectivity of a single node ● d/(sqrt(3)); d/(2sqrt(3)); d/(4sqrt(3));.... ● We call these networks “PLCN” (Physical wire Length Constrained Networks)

20. Intuitive Proof ● Intuitively, anything along the diameter would be easy to reach ● sqrt(2)*max(x,y) hops is the max number of hops ● Once we reach a region by the longest hops, we explore the smaller region recursively

21. Optimal value of 'd' ● For different values of i, hence k, we optimize the value of 'd' which would minimize 'D' ● 1D – For i=1; D=L/d + d/2; d=sqrt(L) ● 2D – For i=1; D=sqrt(2)*L/d +d/sqrt(2); D= d=sqrt(2L); L*L grid ● 3D – For i=1; D=sqrt(3)*L/d +d/sqrt(3); d=sqrt(3L); L*L*L grid ● For i=2, the equations become sufficiently complicated

22. Average shortest all pair hop distance for k=8; 'd'; d=1 and d=4

23. Average shortest all pair hop distance for k=8; d=6 and d=9

24. Routing ● Simple routing algorithm for different dimensions – Use the longest available hop towards the destination – Within this smaller region, use the above step recursively – The final step is to reach it by using simple hops along the lowest level connections along the axis ● With some care, this could be easily coneverted to adaptive minimal routing (if more than one shortest paths are available) ● Non-minimal routing

25. Conclusions ● Communication Topology is important ● Message latency should be minimized for an application to scale ● Non-trivial networks like LDR and PLCN are not so difficult to implement ● Can drastically reduce average all pair shortest distance and diameter