1. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Parallel Programming in C with MPI and OpenMP Michael J. Quinn

2. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Chapter 2 Parallel Architectures

3. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Outline Interconnection networks Interconnection networks Processor arrays Processor arrays Multiprocessors Multiprocessors Multicomputers Multicomputers Flynn’s taxonomy Flynn’s taxonomy

4. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Interconnection Networks Uses of interconnection networks Uses of interconnection networks  Connect processors to shared memory  Connect processors to each other Interconnection media types Interconnection media types  Shared medium  Switched medium

5. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Shared versus Switched Media

6. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Shared Medium Allows only message at a time Allows only message at a time Messages are broadcast Messages are broadcast Each processor “listens” to every message Each processor “listens” to every message Arbitration is decentralized Arbitration is decentralized Collisions require resending of messages Collisions require resending of messages Ethernet is an example Ethernet is an example

7. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Switched Medium Supports point-to-point messages between pairs of processors Supports point-to-point messages between pairs of processors Each processor has its own path to switch Each processor has its own path to switch Advantages over shared media Advantages over shared media  Allows multiple messages to be sent simultaneously  Allows scaling of network to accommodate increase in processors

8. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Switch Network Topologies View switched network as a graph View switched network as a graph  Vertices = processors or switches  Edges = communication paths Two kinds of topologies Two kinds of topologies  Direct  Indirect

9. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Direct Topology Ratio of switch nodes to processor nodes is 1:1 Ratio of switch nodes to processor nodes is 1:1 Every switch node is connected to Every switch node is connected to  1 processor node  At least 1 other switch node

10. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Indirect Topology Ratio of switch nodes to processor nodes is greater than 1:1 Ratio of switch nodes to processor nodes is greater than 1:1 Some switches simply connect other switches Some switches simply connect other switches

11. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Evaluating Switch Topologies Diameter Diameter Bisection width Bisection width Number of edges / node Number of edges / node Constant edge length? (yes/no) Constant edge length? (yes/no)

12. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 2-D Mesh Network Direct topology Direct topology Switches arranged into a 2-D lattice Switches arranged into a 2-D lattice Communication allowed only between neighboring switches Communication allowed only between neighboring switches Variants allow wraparound connections between switches on edge of mesh Variants allow wraparound connections between switches on edge of mesh

13. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 2-D Meshes

14. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Evaluating 2-D Meshes Diameter:  (n 1/2 ) Diameter:  (n 1/2 ) Bisection width:  (n 1/2 ) Bisection width:  (n 1/2 ) Number of edges per switch: 4 Number of edges per switch: 4 Constant edge length? Yes Constant edge length? Yes

15. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Binary Tree Network Indirect topology Indirect topology n = 2 d processor nodes, n-1 switches n = 2 d processor nodes, n-1 switches

16. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Evaluating Binary Tree Network Diameter: 2 log n Diameter: 2 log n Bisection width: 1 Bisection width: 1 Edges / node: 3 Edges / node: 3 Constant edge length? No Constant edge length? No

17. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Hypertree Network Indirect topology Indirect topology Shares low diameter of binary tree Shares low diameter of binary tree Greatly improves bisection width Greatly improves bisection width From “front” looks like k-ary tree of height d From “front” looks like k-ary tree of height d From “side” looks like upside down binary tree of height d From “side” looks like upside down binary tree of height d

18. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Hypertree Network

19. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Evaluating 4-ary Hypertree Diameter: log n Diameter: log n Bisection width: n / 2 Bisection width: n / 2 Edges / node: 6 Edges / node: 6 Constant edge length? No Constant edge length? No

20. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Butterfly Network Indirect topology Indirect topology n = 2 d processor nodes connected by n(log n + 1) switching nodes n = 2 d processor nodes connected by n(log n + 1) switching nodes

21. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Butterfly Network Routing

22. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Evaluating Butterfly Network Diameter: log n Diameter: log n Bisection width: n / 2 Bisection width: n / 2 Edges per node: 4 Edges per node: 4 Constant edge length? No Constant edge length? No

23. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Hypercube Directory topology Directory topology 2 x 2 x … x 2 mesh 2 x 2 x … x 2 mesh Number of nodes a power of 2 Number of nodes a power of 2 Node addresses 0, 1, …, 2 k -1 Node addresses 0, 1, …, 2 k -1 Node i connected to k nodes whose addresses differ from i in exactly one bit position Node i connected to k nodes whose addresses differ from i in exactly one bit position

24. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Hypercube Addressing

25. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Hypercubes Illustrated

26. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Evaluating Hypercube Network Diameter: log n Diameter: log n Bisection width: n / 2 Bisection width: n / 2 Edges per node: log n Edges per node: log n Constant edge length? No Constant edge length? No

27. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Shuffle-exchange Direct topology Direct topology Number of nodes a power of 2 Number of nodes a power of 2 Nodes have addresses 0, 1, …, 2 k -1 Nodes have addresses 0, 1, …, 2 k -1 Two outgoing links from node i Two outgoing links from node i  Shuffle link to node LeftCycle(i)  Exchange link to node [xor (i, 1)]

28. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Shuffle-exchange Illustrated 01234567

29. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Shuffle-exchange Addressing

30. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Evaluating Shuffle-exchange Diameter: 2log n - 1 Diameter: 2log n - 1 Bisection width:  n / log n Bisection width:  n / log n Edges per node: 2 Edges per node: 2 Constant edge length? No Constant edge length? No

31. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Comparing Networks All have logarithmic diameter except 2-D mesh All have logarithmic diameter except 2-D mesh Hypertree, butterfly, and hypercube have bisection width n / 2 Hypertree, butterfly, and hypercube have bisection width n / 2 All have constant edges per node except hypercube All have constant edges per node except hypercube Only 2-D mesh keeps edge lengths constant as network size increases Only 2-D mesh keeps edge lengths constant as network size increases