Auburn University http://www.eng.auburn.edu/~xqin COMP8330/7330/7336 Advanced Parallel and Distributed Computing Interconnection Networks (Part 2) Dr. Xiao Qin Auburn University http://www.eng.auburn.edu/~xqin xqin@auburn.edu Lec02

Recap: Static and Dynamic Interconnection Networks Classification of interconnection networks: (a) a static network; and (b) a dynamic network.

Recap: Bus A collection of parallel communication wires together with hardware controlling the bus. Communication wires are shared by the devices Pros: easy to build, good broadcast Some of the simplest and earliest parallel machines used buses. Cons: As the number of devices connected to the bus increases, contention for use of the bus increases, and performance decreases. However, the bandwidth of the shared bus is a major bottleneck. All processors access a common bus for exchanging data. The distance between any two nodes is O(1) in a bus. The bus also provides a convenient broadcast media. Typical bus based machines are limited to dozens of nodes. Sun Enterprise servers and Intel Pentium based shared-bus multiprocessors are examples of such architectures.

Recap: Crossbars A crossbar network uses an p×m grid of switches to connect p inputs to m outputs in a non-blocking manner. A completely non-blocking crossbar network connecting p processors to b memory banks.

Multistage Interconnection Networks Network Topologies: Multistage Networks Multistage interconnection networks (MINs) are a class of high-speed computer networks usually composed of processing elements (PEs) on one end of the network and memory elements (MEs) on the other end, connected by switching elements (SEs). The switching elements themselves are usually connected to each other in stages, hence the name. Such networks include networks, omega networks, delta networks and many other types. MINs are typically used in high-performance or parallel computing as a low-latency interconnection (as opposed to traditional packet switching networks), though they could be implemented on top of a packet switching network. Though the network is typically used for routing purposes, it could also be used as a co-processor to the actual processors for such uses as sorting; cyclic shifting, as in a perfect shufflenetwork; and bitonic sorting. Looks like a pipeline One of the most commonly used multistage interconnects This network consists of log p stages, where p is the number of inputs/outputs. The schematic of a typical multistage interconnection network.

Multistage Omega Network At each stage, input i is connected to output j if: Q1: What does this expression mean? Each stage of the Omega network implements a perfect shuffle. A perfect shuffle interconnection for eight inputs and outputs. Multistage networks include networks, omega networks, delta networks and many other types.

Multistage Omega Network The perfect shuffle patterns are connected using 2×2 switches. The switches operate in two modes – crossover or passthrough. Draw a black box (2 in and 2 out) representing a switch component Two switching configurations of the 2 × 2 switch: (a) Pass-through; (b) Cross-over. Q2: How many modes does a switch component operate in?

Multistage Omega Network A complete Omega network with the perfect shuffle interconnects and switches can now be illustrated: Q3: If the number of processors is p, what is the cost? Give one minute to compute the cost A complete omega network connecting eight inputs and eight outputs. An omega network has p/2 × log p switching nodes, and the cost of such a network grows as (p log p).

Multistage Omega Network: Routing Let s be the binary representation of the source and d be that of the destination processor. The data traverses the link to the first switching node. If the most significant bits of s and d are the same, then the data is routed in pass-through mode by the switch else, it switches to crossover. This process is repeated for each of the log p switching stages. Note that this is not a non-blocking switch. Give the students two minutes to read this policy, then explain This is a reference slide. Q4: How does this routing policy work?

Multistage Omega Network Routing Walkthrough this example An example of blocking in omega network: one of the messages (010 to 111 or 110 to 100) is blocked at link AB.

Completely Connected and Star Connected Networks Example of an 8-node completely connected network. (a) A completely-connected network of eight nodes; (b) a star connected network of nine nodes.

Completely Connected Network Each processor is connected to every other processor. The number of links in the network scales as? O(p2). While the performance scales very well, the hardware complexity is not realizable for large values of p. In this sense, these networks are static counterparts of crossbars. Q5: What is a problem here?

Star Connected Network Every node is connected only to a common node at the center. Distance between any pair of nodes is? O(1). However, the central node becomes a bottleneck. In this sense, star connected networks are static counterparts of buses. O(1) = O(2) Reference: http://stackoverflow.com/questions/1997262/what-are-the-differences-between-o1-and-o2-in-algorithm-analysis There is no difference between O(1) and O(2). Algorithms classifying as O(1) are O(2) and vice versa. In fact, O(c1) is O(c2) for any positive constants c1 and c2. O(c) where c is a positive constants simply means that the runtime is bounded independent of the input or problem size. From this it is clear (informally) that O(1) and O(2) are equal. Formally, consider a function f in O(1). Then there is a constant c such that f(n) <= c * 1 for all n. Let d = c / 2. Then f(n) <= c = (c / 2) * 2 = d * 2 which shows that f is O(2). Similarly if g is O(2) there is a constant c such that g(n) <= c * 2 for all n. Let d = 2 * c. Then g(n) <= c * 2 = d = d * 1 which shows that g is O(1). Therefore O(1) = O(2). Q6: What is a problem here?

Linear Arrays, Meshes, and k-d Meshes A special case of a d-dimensional mesh is a hypercube. Here, d = log p, where p is the total number of nodes.

Network Topologies: Linear Arrays Linear arrays: (a) with no wraparound links; (b) with wraparound link. In a linear array, each node has two neighbors, one to its left and one to its right. If the nodes at either end are connected, we refer to it as a 1-D torus or a ring.

Two- and Three Dimensional Meshes Two and three dimensional meshes: (a) 2-D mesh with no wraparound; (b) 2-D mesh with wraparound link (2-D torus); and (c) a 3-D mesh with no wraparound. A generalization to 2 dimensions has nodes with 4 neighbors, to the north, south, east, and west. Q7: How many neighbors for 3D? A further generalization to d dimensions has nodes with 2d neighbors.

Hypercubes and their Construction Construction of hypercubes from hypercubes of lower dimension. Q8: How many neighbors? Number of neighbors = D -> log(p)

Properties of Hypercubes The distance between any two nodes is at most log p. (Why?) Each node has log p neighbors. The distance between two nodes is given by the number of bit positions at which the two nodes differ.

Summary Multistage Omega Network Completely Connected Star Connected Networks Linear Arrays, Meshes, and k-d Meshes Hypercubes