Princess Sumaya Univ. Computer Engineering Dept. Chapter 7:

Princess Sumaya University – Computer Arch. & Org (2) Computer Engineering Dept. 1 / 9 Parallelism  Uniprocessor vs. Multiprocessors ●Process per Processor  Process-Level Parallelism ●Parallel Processing Program (Multithreading)  Multicore vs. Cluster ●Single Chip vs. LAN Interconnect

Princess Sumaya University – Computer Arch. & Org (2) Computer Engineering Dept. 2 / 9 Parallel Processing Program  Amdahl’s Law Exercise: To achieve a speedup of 90 times faster with 100 processors, what percentage of the original computation can be sequential? Execution Time Execution Time Affected Execution After = ──────────────── + Time Improvement Amount of Improvement Unaffected

Princess Sumaya University – Computer Arch. & Org (2) Computer Engineering Dept. 3 / 9 Scaling  Strong Scaling Speedup achieved on a multiprocessor without increasing the size of the problem. Exercise: Consider sum of 10 scalars (10 sequential additions, T add ) and sum of two 10 × 10 matrixes (100 parallel additions). What are the speedups for 10 & 100 processors?

Princess Sumaya University – Computer Arch. & Org (2) Computer Engineering Dept. 4 / 9 Scaling  Weak Scaling Speedup achieved on a multiprocessor while increasing size of the problem proportional to increase in # of processors. Exercise: Consider sum of 10 scalars (10 sequential additions, T add ) and sum of two 100 × 100 matrixes (10,000 parallel additions). What are the speedups for 10 & 100 processors?

Princess Sumaya University – Computer Arch. & Org (2) Computer Engineering Dept. 5 / 9 Load Balance  Non Ideal Balance Processors don’t get equal amount of work. Exercise: Consider 10 sequential additions and 10,000 parallel additions using 100 processors. What is the speedup when a processor has 2% of the load instead of 1%? What about 5% of the load?

Princess Sumaya University – Computer Arch. & Org (2) Computer Engineering Dept. 6 / 9 Shared Memory Multiprocessors (SMP)  Single Physical Address Space ●Uniform Memory Access (UMA) ●Nonuniform Memory Access (NUMA) ●Synchronization (Lock) Main Memory ● ● ● InterconnectInterconnect CacheCache ProcessorProcessor I/O Controller CacheCache ProcessorProcessor CacheCache ProcessorProcessor ● ● ●

Princess Sumaya University – Computer Arch. & Org (2) Computer Engineering Dept. 7 / 9 Message-Passing Multiprocessors  Private Physical Address Space ●Send-Message & Receive-Message Routines Main Memory ● ● ● InterconnectInterconnect CacheCache ProcessorProcessor I/O Controller CacheCache ProcessorProcessor CacheCache ProcessorProcessor ● ● ● Main Memory ● ● ●

Princess Sumaya University – Computer Arch. & Org (2) Computer Engineering Dept. 8 / 9 Multithreading  Hardware Multithreading ●Sharing Processor’s Functional Units Among Threads (Switch state from one thread to another when stalled)  Fine-Grained Multithreading ●Switching State After Every Instruction  Coarse-Grained Multithreading ●Switching State After a Cache Miss  Simultaneous Multithreading (SMT) ●Multiple-Issue, Dynamically Scheduled Processor (Exploits thread-level & instruction-level parallelism)

Princess Sumaya University – Computer Arch. & Org (2) Computer Engineering Dept. 9 / 9 SISD, MIMD, SIMD, SPMD  Single-Instruction Single-Data ●Uniprocessor  Multiple-Instruction Multiple-Data ●Multiprocessor  Single-Instruction Multiple-Data ●Vector/Array Processor (Data-Level Parallelism)  Single-Program Multiple-Data ●Different Code Sections Execute in Parallel (MIMD)

Princess Sumaya University – Computer Arch. & Org (2) Computer Engineering Dept. Chapter 7