1. Dynamic Interconnection Networks

2. Overview Network properties Switches
Single and multistage Interconnection networks
Crossbar

3. Network properties
Node degree d - the number of edges incident on a node.
In degree
Out degree
Diameter D of a network is the maximum shortest path between any two nodes.
The network is symmetric if it looks the same from any node.
The network is scalable if it expandable with scalable performance when the machine resources are increased.
Node degree should be small and constant
D is measured by the number of links traversed
Symmetric networks are easier to implement or program
High Diameter is bad since they make the system not scalable. For example a Linear array has high diameter and the diameter increases proportional to the number of nodes.

4. Bisection width
Bisection width is the minimum number of wires that must be cut to divide the network into two equal halves.
Small bisection width -> low bandwidth
A large bisection width -> a lot of extra wires
A cut of a network C(N1,N2) is a set of channels that partition the set of all nodes into two disjoint sets N1 and N2. Each element of C(N1,N2) is a channel with a source in N1 and destination in N2 or vice versa.
A bisection of a network is a cut that partitions the entire network nearly in half, such that |N2|≤|N1|≤|N2+1|. Here |N2| means the number of nodes that belong to the partition N2.
The channel bisection of a network is the minimum channel count over all bisections of the network:
The Bisection width tell us about the bandwidth of the network.

5. Factors Affecting Performance
Functionality – how the network supports data routing, interrupt handling, synchronization, request/message combining, and coherence
Network latency – worst-case time for a unit message to be transferred
Bandwidth – maximum data rate
Hardware complexity – implementation costs for wire, logic, switches, connectors, etc.
Bandwidth is an average number of requests that can be accepted by a network

6. 2 × 2 Switches The different settings of a 2x2 Switching Element
*From Advanced Computer Architectures, K. Hwang, 1993.

7. Switches
Module size
Legitimate states
Permutation connection
2 × 2 4 2
4 × 4
256 24
8 × 8
16,777,216
40,320
N × N NN N!
Permutation function: each input can only be connected a single output.
Legitimate state: Each input can be connected to multiple outputs, but each output can only be connected to a single input
Shifting, permutation, broadcast, shuffle

8. Single-stage networks
Single stage Shuffle-Exchange IN (left)
Perfect shuffle mapping function (right)
Perfect shuffle operation: cyclic shift 1 place left, eg > 011
Exchange operation: invert least significant bit, e.g > 100
In single stage networks there are different methods needed to change the interconnection pattern so that a given input
is sent to the desired output. However with one shuffle or exchange operation the desired output might not be given so multiple shuffles will need to occur. Which is why a multistage network was build.
*From Ben Macey at

9. Multistage Interconnection Networks
The capability of single stage networks are limited but if we cascade enough of them together, they form a completely connected MIN (Multistage Interconnection Network).
Switches can perform their own routing or can be controlled by a central router
This type of networks can be classified into the following four categories:
Nonblocking
A network is called strictly nonblocking if it can connect any idle input to any idle output regardless of what other connections are currently in process
Rearrangeable nonblocking
In this case a network should be able to establish all possible connections between inputs and outputs by rearranging its existing connections.
Blocking interconnection
A network is said to be blocking if it can perform many, but not all, possible connections between terminals.
Example: the Omega network
Self-routing network. Since a single-stage network was not sufficient to get the desired output, a multistage is built. An example of a multistage network is the omega network.

10. Omega networks
A multi-stage IN using 2 × 2 switch boxes and a perfect shuffle interconnect pattern between the stages
In the Omega MIN there is one unique path from each input to each output.
No redundant paths → no fault tolerance and the possibility of blocking
Example:
Connect input 101 to output 001
Use the bits of the destination address, 001, for dynamically selecting a path
Routing:
- 0 means use upper output
- 1 means use lower output
Isomorphic networks can be easily reproduced from the other networks by simply rearranging nodes at each stage.
5->1 and 1->3
*From Ben Macey at

11. Omega networks log2N stages of 2 × 2 switches N/2 switches per stage
S=(N/2) log2(N) switches
Number of permutations in a omega network 2S
Even though this type of network allows for input/output paths to be independently
determined by the switches, blocking can still occur in an Omega network if two “packets” arrive at the same switch at the same time.

12. Baseline networks The network can be generated recursively
The first stage N × N, the second (N/2) × (N/2)
Networks are topologically equivalent if one network can be easily reproduced from the other networks by simply rearranging nodes at each stage.
the inter-stage connections are created recursively using the method of cyclic bit shifting.
The bit representation of an input is shifted right by 1 bit to make the output number.
Between stages the next stage is always cut into 2 separate sections. So the second stage would have 2 sections and the third stage would have 4 sections and so on.
The switch rules are similar to Omega networks.
*From Advanced Computer Architectures, K. Hwang, 1993.

13. Crossbar Network
Each junction is a switching component – connecting the row to the column.
Can only have one connection in each column
The crossbar is a special type of dynamic interconnection network.
Every input has route to every output directly, eliminating blocking.
A crossbar network offers better performance over other
The trade-off here is a crossbar design requires more switching elements than other network types, increasing cost.
*From Advanced Computer Architectures, K. Hwang, 1993.

14. Crossbar Network
The major advantage of the cross-bar switch is its potential for speed.
In one clock, a connection can be made between source and destination.
The diameter of the cross-bar is one.
Blocking if the destination is in use
Because of its complexity, the cost of the cross-bar switch can become the dominant factor for a large multiprocessor system.
Crossbars can be used to implement the a×b switches used in MIN’s. In this case each crossbar is small so costs are kept down.

15. Problem
Use two-input AND and OR gates to construct NxN crossbar switch network between N processors and N memory modules. Use cij signal as the enable signal for the switch in ith row and jth column. Let the width of each crosspoint be w bits.
Estimate the total number of AND and OR gates needed as a function of N and w.
Hard to put what I drew on my paper, I can give you a photocopy of my sketch.

16. Problem (cont.)
The crosspoint shown here is only uni directional to simplify the sketch. Next slide shows what the crosspoint looks for birectional (PE -> Mem, Mem -> PE). One Thing to notice here is that the last row does not need an OR gate.

17. Problem (cont.)
Bidirectional crosspoint.

18. Problem (cont.)
A simplify version of the arbiter. This is for a 2 x 2 crossbar. The crossbar uses priority to determine who gets to go first when Two PE try to communicate with a single memory. P1 has priority over P2 , P2 over P3, PN-1 over PN. Cij is the control signals to determine which crosspoint gets “activated”. The decoder gets an address (to determine which memory the PE wants to communicate with). So for example, if P1 wants to communicate with M1, it would send 1 to C11 and C21 would get 0 (since there is a NOT gate). What that means if P2 wanted to communicate with M1 it would have to wait for P1 to finish, since C21 is not activated.

19. Performance Comparison
Network
Latency
Switching
complexity
Wiring
Blocking
Bus
Constant O(N)
O(1)
O(w)
yes
MIN
O(log2N)
O(Nlog2N)
O(Nw log2 N)
Crossbar
O(N2)
O(N2w) no
The complexity of various issues for Interconnecting networks
Using the O-notation to analyze.
From the table, we can observe different trade-offs between designs.
Some more characteristics which must be considered when choosing a design are bandwidth, node degree, diameter and level of symmetry.

20. Some Commercial Solutions [3]
System-on-chip crossbar networks:
Nexus from Fulcrum Microsystems
The core is used in PMC-Sierra dual MIPS processor RM9000
An example using a crossbar on SoC.

21. References
Advanced Computer Architecture and Parallel Processing, by Hesham El-Rewini and Mostafa Abd-El-Barr, John Wiley and Sons, 2005.
Advanced Computer Architecture Parallelism, Scalability, Programmability, by  K. Hwang, McGraw-Hill 1993.
A. Lines, “Nexus: an asynchronous crossbar interconnect for synchronous system-on-chip designs”, Proc. of High Performance Interconnects, pp 2-7, 2003.