Multi-Processing in High Performance Computer Architecture:

What is Multiprocessing? Enables several programs to run concurrently Coordinated processing of Programs by more than one processor Use of 2 or more CPUs within a single computer system Ability of a system to support more than one processor and to allocate tasks between them

Idealism (Target for Processor Performance):

Memory Hierarchies:

Memory in Modern Processor (L1 Cache?):

Why Not Uni-Processors: Making a wider Processor can efficiently run parallel programs but not programs that have dependencies In Uni-Processor, the instructions a = b + c; d = e + f can be run in parallel because non of the results depend on other calculations The instructions b = e + f; a = b + c, might not run in parallel due to data dependencies. There are several more complex data dependencies. Parallel part : Fast, One at a time(Stalls) : Slow

Why Multi-Processors:

Multi-Processor needs Parallel Programs(Years to Develop):

Centralized Shared Memory: (UMA) Distance from memory to each core is approximately same(UMA) Replicate Core and Caches to build a Multi Dimensional System(SMP)

Issues in Centralized Main Memory:

Distributed Memory: Each Core has it’s own local memory and cache forming a single core computer system. A network interface card, connected to a network of interconnections Cache miss goes directly to the local processor’s memory and if needed it can only access the other processor’s memory through the network message passing. To communicate, data is sent EXPLICITLY to the core. Programmer is forced to be aware of communication between cores and try to minimize it. Think of it as a set of machines communicating over a network.

Distributed (Multicomputer) Memory System(NUMA):

Shared Memory vs Message Passing (Hardware vs Software):

Simple Implementation of Summing Array in Parallel on 4 cores (Message Passing):

A Simple Shared Memory Implementation:

Performance Metrics (Message Passing vs Shared Memory):

Multithreading as Shared Memory Hardware:

Analyzing Multithreading Performance:

Summary: Multithreaded Categories