1. Scaling Area Under a Curve

2. Why do parallelism? Speedup – solve a problem faster. Accuracy – solve a problem better. Scaling – solve a bigger problem.

3. Problem 1: communication overhead Parallel algorithms that employ distributed memory have a certain amount of communication and a certain amount of computation. As more processes are added, the communication overhead increases. At some number of processes, communication overhead will cause the algorithm to take longer in parallel than serially.

4. Problem 2: Amdahl’s law Speedup is limited by the algorithm’s critical section – the part of the algorithm that cannot be parallelized. An algorithm will never be faster than its critical section. Speedup = where P = the proportion of the program that can be made parallel 1 – P = the proportion of the program that cannot be made parallel N = the number of processors

5. Strong scaling Strong scaling – increasing the number of processes but keeping the problem size constant.

6. The solution Don’t solve a problem faster, instead solve a bigger problem.

7. Gustafson’s law A bigger problem solved with more processors can be solved in the same amount of time it takes to solve a smaller problem with fewer processors. Speedup(N) = N – (1 – P) * (N – 1) where N = the number of processors (1 – P) = the proportion of the program that cannot be made parallel

8. Weak scaling Weak scaling – increasing the problem size as the number of processes increases.