1. CS252/Patterson Lec 12.1 2/28/01 CS162 Computer Architecture Lecture 16: Multiprocessor 2: Directory Protocol, Interconnection Networks

2. CS252/Patterson Lec 12.2 2/28/01 Larger MPs Separate Memory per Processor Local or Remote access via memory controller 1 Cache Coherency solution: non-cached pages Alternative: directory per cache that tracks state of every block in every cache –Which caches have a copies of block, dirty vs. clean,... Info per memory block vs. per cache block? –PLUS: In memory => simpler protocol (centralized/one location) –MINUS: In memory => directory is ƒ(memory size) vs. ƒ(cache size) Prevent directory as bottleneck? distribute directory entries with memory, each keeping track of which Procs have copies of their blocks

3. CS252/Patterson Lec 12.3 2/28/01 Distributed Directory MPs

4. CS252/Patterson Lec 12.4 2/28/01 Network Examples Bi-directional Ring – EX: HP V Class 2-D Mesh and Hypercube – SGI Origin and Cray T3E Crossbar and Omega Network – SMPs, IBM SP3, and IP Routers Clusters using ethernet, Gigabit ethernet, Myrinet, etc. Properties of various networks will be discussed later

5. CS252/Patterson Lec 12.5 2/28/01 CC-NUMA Multiprocessor: Directory Protocol What is Cache Coherent Non-Uniform Memory Access (CC-NUMA)? Similar to Snoopy Protocol: Three states –Shared: ≥ 1 processors have data, memory up-to-date –Uncached (no processor hasit; not valid in any cache) –Exclusive: 1 processor (owner) has data; memory out-of-date In addition to cache state, must track which processors have data when in the shared state (usually bit vector, 1 if processor has copy) Directory Size: Big => Limited Directory Schemes (Not to be discussed)

6. CS252/Patterson Lec 12.6 2/28/01 Directory Protocol No bus and don’t want to broadcast: –interconnect no longer single arbitration point –all messages have explicit responses Terms: typically 3 processors involved –Local node where a request originates –Home node where the memory location of an address resides –Remote node has a copy of a cache block, whether exclusive or shared Example messages on next slide: P = processor number, A = address

7. CS252/Patterson Lec 12.7 2/28/01 Example Directory Protocol Message sent to directory causes two actions: –Update the directory –More messages to satisfy request Block is in Uncached state: the copy in memory is the current value; only possible requests for that block are: –Read miss: requesting processor sent data from memory &requestor made only sharing node; state of block made Shared. –Write miss: requesting processor is sent the value & becomes the Sharing node. The block is made Exclusive to indicate that the only valid copy is cached. Sharers indicates the identity of the owner. Block is Shared => the memory value is up-to-date: –Read miss: requesting processor is sent back the data from memory & requesting processor is added to the sharing set. –Write miss: requesting processor is sent the value. All processors in the set Sharers are sent invalidate messages, & Sharers is set to identity of requesting processor. The state of the block is made Exclusive.

8. CS252/Patterson Lec 12.8 2/28/01 Example Directory Protocol Block is Exclusive : current value of the block is held in the cache of the processor identified by the set Sharers (the owner) => three possible directory requests: –Read miss: owner processor sent data fetch message, causing state of block in owner’s cache to transition to Shared and causes owner to send data to directory, where it is written to memory & sent back to requesting processor. Identity of requesting processor is added to set Sharers, which still contains the identity of the processor that was the owner (since it still has a readable copy). State is shared. –Data write-back: owner processor is replacing the block and hence must write it back, making memory copy up-to-date (the home directory essentially becomes the owner), the block is now Uncached, and the Sharer set is empty. –Write miss: block has a new owner. A message is sent to old owner causing the cache to send the value of the block to the directory from which it is sent to the requesting processor, which becomes the new owner. Sharers is set to identity of new owner, and state of block is made Exclusive.

9. CS252/Patterson Lec 12.9 2/28/01

10. CS252/Patterson Lec 12.10 2/28/01 Interconnection Network Routing, Topology Design Trade-offs

11. CS252/Patterson Lec 12.11 2/28/01 Interconnection Topologies Class networks scaling with N Logical Properties: –distance, degree Physical properties –length, width Static vs. Dynamic Networks Fully connected network –diameter = 1 –degree = N –cost? »bus => O(N), but BW is O(1) - actually worse »crossbar => O(N 2 ) for BW O(N) VLSI technology determines switch degree

12. CS252/Patterson Lec 12.12 2/28/01 What characterizes a network? Topology (what) –physical interconnection structure of the network graph –direct: node connected to every switch –indirect: nodes connected to specific subset of switches Routing Algorithm(which) –restricts the set of paths that msgs may follow –many algorithms with different properties »gridlock avoidance? Switching Strategy(how) –how data in a msg traverses a route –circuit switching vs. packet switching Flow Control Mechanism(when) –when a msg or portions of it traverse a route –what happens when traffic is encountered?

13. CS252/Patterson Lec 12.13 2/28/01 Flow Control What do you do when push comes to shove? –Ethernet: collision detection and retry after delay –FDDI, token ring: arbitration token –TCP/WAN: buffer, drop, adjust rate –any solution must adjust to output rate Link-level flow control

14. CS252/Patterson Lec 12.14 2/28/01 Topological Properties Routing Distance - number of links on route Diameter - maximum routing distance between any two nodes in the network Average Distance – Sum of distances between nodes/number of nodes Degree of a Node – Number of links connected to a node => Cost high if degree is high A network is partitioned by a set of links if their removal disconnects the graph Fault-tolerance – Number of alternate paths between two nodes in a network

15. CS252/Patterson Lec 12.15 2/28/01 Review: Performance Metrics Sender Receiver Sender Overhead Transmission time (size ÷ bandwidth) Transmission time (size ÷ bandwidth) Time of Flight Receiver Overhead Transport Latency Total Latency = Sender Overhead + Time of Flight + Message Size ÷ BW + Receiver Overhead Total Latency (processor busy) (processor busy) Includes header/trailer in BW calculation?

16. CS252/Patterson Lec 12.16 2/28/01 Example Static Network: 2-D Mesh Architecture

17. CS252/Patterson Lec 12.17 2/28/01 More Static Networks: Linear Arrays and Rings Linear Array –Diameter? –Average Distance? –Bisection bandwidth? –Route A -> B given by relative address R = B-A Torus? Examples: FDDI, SCI, FiberChannel Arbitrated Loop, KSR1

18. CS252/Patterson Lec 12.18 2/28/01 Multidimensional Meshes and Tori d-dimensional array –n = k d-1 X...X k O nodes –described by d-vector of coordinates (i d-1,..., i O ) d-dimensional k-ary mesh: N = k d –k = d  N –described by d-vector of radix k coordinate d-dimensional k-ary torus (or k-ary d-cube)? Ex: Intel Paragon (2D), SGI Origin (Hypercube), Cray T3E (3DMesh) 2D Grid 3D Cube

19. CS252/Patterson Lec 12.19 2/28/01 Hypercubes Also called binary n-cubes. # of nodes = N = 2 n. O(logN) Hops Good bisection BW Complexity –Out degree is n = logN correct dimensions in order –with random comm. 2 ports per processor 0-D1-D2-D3-D 4-D 5-D !

20. CS252/Patterson Lec 12.20 2/28/01 Origin Network Each router has six pairs of 1.56MB/s unidirectional links –Two to nodes, four to other routers –latency: 41ns pin to pin across a router Flexible cables up to 3 ft long Four “virtual channels”: request, reply, other two for priority or I/O

21. CS252/Patterson Lec 12.21 2/28/01 Case Study: Cray T3D Build up info in ‘shell’ Remote memory operations encoded in address

22. CS252/Patterson Lec 12.22 2/28/01 Trees Diameter and ave distance logarithmic –k-ary tree, height d = log k N –address specified d-vector of radix k coordinates describing path down from root Fixed degree Route up to common ancestor and down –R = B xor A –let i be position of most significant 1 in R, route up i+1 levels –down in direction given by low i+1 bits of B H-tree space is O(N) with O(  N) long wires Bisection BW?

23. CS252/Patterson Lec 12.23 2/28/01 Real Machines Wide links, smaller routing delay Tremendous variation MachineTopologyCycle Time (ns) Channel Width (bits) Routing Delay (cycles) Flit (data bits) nCUBE/2Hypercube2514032 TMC CM-5Fat-Tree254104 IBM SP-2Banyan258516 Intel Paragon2D Mesh11.5162 Meiko CS-2Fat-Tree20878 CRAY T3D3D Torus6.67162 DASHTorus30162 J-Machine3D Mesh31828 MonsoonButterfly20162 SGI OriginHypercube2.52016160 MyricomArbitrary6.25165016