1. Computer Architecture Distributed Memory MIMD Architectures Ola Flygt Växjö University http://w3.msi.vxu.se/users/ofl/ Ola.Flygt@msi.vxu.se +46 470 70 86 49

2. Outline  Definition and Design Space  Computational Model  Granularity  Node organization  Interconnection network  Topology  Switching  Routing CH01

3. Multicomputers  Distributed Memory MIMD systems are often called Multicomputers  They may or may not have a virtually shared address space  They are typically more loosely coupled than Shared Memory MIMDs

4. Design space of Multicomputers CH01

5. Computational Model  In theory any computational model may be used  In practice some have been implemented  Conventional + communication  CSP (Communicating Sequential Processes)  Dataflow or actor based object oriented  Today almost only the conventional model is used

6. Granularity  As before Granularity is a parameter which applies to both node size and how much the problem is partitioned  They are of course interrelated and as before we have some options  Fine grained  Medium grained  Coarse grained

7. Generic Node Architecture CH01

8. Generic Organization Model Of The Message-Passing Multicomputers 1 st generation CH01

9. Generic Organization Model Of The Message-Passing Multicomputers Decentralized 2 nd generation CH01

10. Generic Organization Model Of The Message-Passing Multicomputers Centralized 2 nd generation CH01

11. Generic Organization Model Of The Message-Passing Multicomputers 3 rd generation CH01

12. Classification Of Multicomputers CH01

13. Main Network Topologies  Linear array  Ring  Star  Binary tree  Fat tree  2-D mesh  2-D wraparound mesh  Hypercube  3-cube connected cycle  Completely connected CH01

14. Main Network Topologies CH01

15. Main Network Topologies CH01

16. Main Network Topologies CH01

17. Static Network parameters TopologyNode degreeDiameter Bisection width Arc connectivityCost Linear array1 or 2N-111 Ring2N/222N Star1 or N-1211N-1 Binary tree1, 2 or 3 2log((N+1)/2 )11N-1 2-D mesh2, 3 or 42(N½-1)N½22(N-N½) 2-D wraparound mesh4N½ 2N½ 42N 3-D cube3, 4, 5 or 63(N ⅓ -1)N⅔N⅔ 32(N-N ⅔ ) HypercubelogN N/2logN(NlogN)/2 Completely connectedN-11N 2 /4N-1N(N-1)/2 CH01

18. Design Space of Switching techniques CH01

19. Packet Switching arrangement CH01

20. Packet Switching latency CH01

21. Packet Switching  All the messages are divided into packets which are sent independently via the communication network between the source and destination nodes  The messages are transmitted in a store-and- forward fashion (each byte contained in a message had to be stored at each node along a route and forwarded to the next hope.) CH01

22. Packet Switching  A packet consists of a header and the data. A header contains the necessary routing information based on that the switching unit decides where to forward the packet  When a packet arrives at an intermediate node, the whole packet is stored in a packet buffer.  Main drawback: latency is proportional to the message path length. (This was the reason why the diameter was the most important parameter in the first generation multicomputers, and why the hypercube was so popular. CH01

23. Circuit Switching arrangement CH01

24. Circuit Switching latency CH01

25. Circuit Switching  In the first phase of the communication a path (a circuit) is built up between the source and destination by sending a special short message (called probe)  The probe has similar function as the header of packets in the packet switching system  The circuit is held until the entire message is transmitted  During the communication the channels constituting the circuit are reserved exclusively, no other messages can be transmitted by them. CH01

26. Circuit Switching  In the last phase the circuit is torn down either by the tail of transmitted message or by an acknowledgement message returned by the destination node  If a desired channel is used by another circuit in the circuit establishment phase, the partially built up circuit may be torn down  Circuit switching does not need packetizing. No matter what the message size is  There is no need for buffering  The most important benefit: If the length of the probe, P is much smaller then the length of the message, M then the latency becomes independent of the communication distance. CH01

27. Virtual Cut-Through Switching arrangement CH01

28. Virtual Cut-Through Switching latency CH01

29. Virtual Cut-Through Switching  Attempt to combine the benefits of packet switching and circuit switching  The message is divided in to small units called flow control digits, or flits  As long as the required channels are free, the message is forwarded flit by flit among the nodes in a pipeline fashion  If the required channel is busy, flits are buffered at intermediate nodes CH01

30. Virtual Cut-Through Switching  If the buffers are large enough, the entire message is buffered at the blocked intermediate node, resulting a behavior similar to packet switching  If the buffers are not large enough, the message will be buffered across several nodes, holding the links between them  Main benefit: If HF (length of the header flit) << M, the latency becomes independent of the distance, D CH01

31. Worm-Hole Switching arrangement CH01

32. Worm-Hole Switching latency CH01

33. Worm-Hole Switching  Do not create a circuit between sender and receiver. Instead, an initial control message at the start of the message establishes a path through the network and all subsequent data for that message are forwarded along that path  The message is broken into very small pieces (flits) and the network is pipelined. This is referred to as worm-hole routing due to the way that a message worms its way through the system CH01

34. Worm-Hole Switching  A special case of virtual cut-through, where the buffers at the intermediate nodes have the size of a flit  There is a no start-up overhead related to distance, the entire message is not penalized due to the pipelining. If P (packet size) is small relative to N (message length), T will be similar to that for the circuit switching system in that T is not very dependent on D (distance)  The primary advantage of such a network is that links need not be blocked for the entire message duration, and (after introducing the virtual channel concept) it is possible to multiplex messages along individual links. CH01

35. Routing protocols  Location of routing ”intelligence’’  Source-based routing  Routers “eat” the head of a packet  Larger packets  No fault tolerance  Distributed (Local) routing  More complex routers  Smaller packets

36. Classification of Routing protocols CH01

37. Classification of Adaptive Routing protocols CH01

38. Routing protocols Terminology  Minimal = only paths equal to the shortest path is selected  Profitable = only channels known to move closer to the goal is selected  Misrouting = all channels may be used  Progressive = never backtracks even if blocked  Partially adaptive = not all channels may be selected as the next step CH01

39. Routing protocols  Routing may cause:  Deadlocks  Buffer deadlock (store-and-forward switching)  Channel deadlock (wormhole routing)  Livelocks  Packets are forwarded in a loop in the network

40. Routing deadlocks CH01

41. Routing protocols Deterministic routing  X-Y Routing  Walk one dimension at a time CH01

42. Routing protocols Deterministic routing  Interval labeling  Distributed routing with simple routing tables in the nodes CH01

43. Routing protocols Adaptive routing  Decision on next channel based on the current blocking situation  Can potentially give better utilization CH01

44. Deadlock avoidance  Deterministic routing (e.g. X-Y)  Partially adaptive routing  For example, west-first routing for 2D meshes: route a packet first to the west (if required), then route the packet adaptively to north, south or east CH01

45. West-first routing example CH01

46. Deadlock avoidance, cont.  Virtual channels  Virtual channels are logical links between two nodes using their own buffers and multiplexed over a single physical channel  Virtual channels “break” dependency cycles CH01

47. Virtual channels  Advantages  Increased network throughput  Deadlock avoidance  Virtual topologies  Dedicated channels (e.g. debugging, monitoring)  Disadvantages  Hardware cost  Higher latency  Incoming packets may be out-of-order CH01

48. Complex communication support  Common communication patterns  Partner communication (unicast)  Multicast (one-to-many)  Broadcast (one-to-all)  Exchange (many-to-many or all-to-all)  Routers may feature hardware support for these communication patterns CH01

49. Complex communication support  An example: multicast support  Software unicasts often implemented using a tree communication structure  “Replication” and “Routing support” require the destinations in the packet-header  Replication often found in wormhole networks CH01

50. Multicomputers today  The idea with a Distributed Memory MIMD is the basis for most parallel (super computer) systems today  The idea have evolved into  Cluster computing, using a LAN as interconnection network  Grid computing, using more loosely connected nodes (WAN, different owners) CH01