1. Introduction to
Parallelism

2. What is Parallelism? Program is split among numerous “entities” Run:
Multiple tasks of program simultaneously
Multiple portions of data calculated simultaneously
Serial vs. Parallel

3. Types of Parallelism Distributed Memory Shared Memory
Hybrid Distributed + Shared Memory

4. Hardware for Parallelism
CPU
Classic model  1 CPU connected to memory
Parallel model  “multi-core” splits CPU into cores (compute units)
GPU (Graphics Processing Unit)
Discuss more next week

5. Why Parallelism? SoonerSpeedup
Program will run fasters if it is parallelized as opposed to executing serially
Allows a problem to be modeled faster
BetterAccuracy
Forming a better solution to the problem
More processes assigned to a tasks  spend more time to do error checks  final result is a better approximation
To make a program more accurate, speedup may be sacrificed
MoreScaling
More parallel processors can be used to model a bigger problem in the same amount of time as fewer parallel processors to model a smaller problem
Fishing analogy

6. Gustafson’s Law
Bigger problems can be modeled in the same amount of time as smaller problems if the processor count is increased
where
N = number of processors
P = the proportion of the program that cannot be made parallel

7. Problems with Parallelism
Communication Overhead
Time lost waiting for communications to take place before and after calculations
More processors is not necessarily better (find a balance)
Can make program less efficient than serial execution
Amdahl’s Law
Speedup of a parallel program is limited by serial regions or regions that cannot be executed in parallel
where
P = proportion of the program that can be made parallel
1 – P = proportion of the program that cannot be made parallel
N = number of processors

8. Example: Image Processing