Copyright © 2011 Curt Hill MIMD Multiple Instructions Multiple Data

Multiprocessors Tightly coupled CPUs –Shared memory system –Share virtual memory as well Any threaded program operates in this mode Communication is usually through memory Example: –Word typing with one thread –Examining spelling in another Copyright © 2011 Curt Hill

Interconnections If the number of processors is small the bus is the interconnection device Small is typically less than When no longer small contention for the bus will reduce performance For larger sizes there must be separate buses and interconnections between them Copyright © 2011 Curt Hill

Cache Complications Even if the memory is truly shared such as on a bus we have problems with caches The problem is that two CPUs have two different caches and that the cache lines are different for the same memory location –Easy to do with write back policy –Not that hard with write through Copyright © 2011 Curt Hill

Cache Coherence Coherent caches disallow different cache lines for same memory locations There are several cache coherence protocols but the results should be similar –The caches agree with each other when they possess the same memory location One such protocol is called MESI Copyright © 2011 Curt Hill

Multiple memories With multiple bus interconnections each CPU will have memory This memory is to be shared with all the others The memories are patched together to provide the illusion that it is just one This can be done with a variety of switches –Crossbar or multistage Copyright © 2011 Curt Hill

Memory Consistency Types Strict – no cache, truly shared Sequential – all processors observe the same order Processor – writes are always seen in the same order Weak – does not guarantee writes in same order Release – serialized weak consistency Copyright © 2011 Curt Hill

Memory Hierarchies There is a limit how many CPUs may be added using the various switches Hardware costs increase as well NonUniform Memory Access NUMA architectures have quicker access to local memory and slower to non-local Cache coherent NUMAs complicate things even more Copyright © 2011 Curt Hill

Multicomputers Often loosely coupled CPUs –Always has some private memory –May have shared memory via a portion of their virtual memory space –Distributed memory system Must pass messages via high speed networking Allows larger scaling than multiprocessors Copyright © 2011 Curt Hill

MPP Massively Parallel Processors Typically commodity CPUs, such as Pentiums –Hundreds or thousands of them Very high-speed interconnection network –Low latency, high bandwidth message passing Very large I/O capabilities –Massive computing needs massive data Copyright © 2011 Curt Hill

Current King As of 11/2011 Japan’s K computer was on top of the super computer list Contains 705,024 SPARC64 cores Reaches 10 Petaflops/sec –10 quadrillion floating point operations a second Copyright © 2011 Curt Hill

Cray XT5 A Cray XT5 at Oak Ridge yielded 1.7 –Largest US super computer It was the king in 2009 Uses 37,333 AMD Opterons, each with 6 cores Copyright © 2011 Curt Hill

Tianhe Ia Currently second on Top 500 list 14,336 Xeon X5670 processors and 7,168 Nvidia Tesla M2050 GPUs 2.57 Petaflops Copyright © 2011 Curt Hill

Cluster Computing Most cluster computing schemes use ordinary workstations They communicate over conventional LAN/WAN This is the software alternative to the Multicomputers Beowulf cluster is an example Copyright © 2011 Curt Hill

Grid Computing Similar to cluster, but the running software does not dominate the machine Typically there is a central server that assigns tasks and receives results When the workstation is idle the grid program consumes the idle cycles Super computing for those with no budget Copyright © 2011 Curt Hill

Examples BOINC – Berkely Open Infrastructure for Network Computing –Client for a variety of different projects –5.6 Petaflops – Protein folding –5 Petaflops – Search for Extra Terrestial Intelligence –730 TeraFlops Copyright © 2011 Curt Hill