1. Parallel Processing
Sharing the load

2. Inside a Processor Primarily Crystalline Silicon
1 mm – 25 mm on a side
100 million to billions of transistors
current “feature size” (process) ~ 14 nanometers
Package provides:
communication with motherboard
heat dissipation
Chip in Package
Circuits

3. Moore's Law Number of transistors in same area doubles every 2 years
Net effects: Processing power doubles approximately every 18 months

4. Exponential Growth Doubling is exponential growth Year Speed 1 2 4 8
1.5 3
4.5 6
7.5 9
10.5 12
Speed 1 2 4 8 16 32 64
128
256

5. Moore's Law If Moore's Law applied to airplanes:
New York to Paris in 1978 $900 & 7 hours

6. Moore's Law If Moore's Law applied to airplanes:
New York to Paris in 1978 $900 & 7 hours
Now should be $0.01 & < 1 second

8. Power Density Prediction circa 2000
4004
8008
8080
8085
8086
286
386
486
Pentium® proc P6 1 10
100
1000
10000
1970
1980
1990
2000
2010
Year
Power Density (W/cm2)
Hot Plate
Nuclear Reactor
Rocket Nozzle
Source: S. Borkar (Intel)
Sun’s Surface
Core 2

9. Going Multi-core Helps Energy Efficiency
Speed takes power, Power = heat
Can run at 80% speed with 50% power

10. MultiCore Multicore : Multiple processing cores on one chip
Each core can run a different program

11. Moore's Law Related Curves

12. Moore's Law Related Curves

13. Issues Not every part of a problem scales well
Parallel : can run at same time
Serial : must run one at a time in order

14. Speedup Issues 5 workers can do parallel portion in 1/5th the time
Can't affect serial part
Time
Parallel portion
Serial portion 1 5
Number of Cores

15. Speedup Issues Increasing workers provide diminishing returns 4 5
Time
Parallel portion
Serial portion 1 2 3 4 5
Number of Cores
Increasing workers provide diminishing returns

16. Amdahl’s Law
Amdahl’s law : Predicts how many times faster N workers can do a task in which P portion is parallel

17. Amdahl’s Law 60% of a job can be made parallel. We use 2 processors:
1.43x faster with 2 than 1

18. Amdahl’s Law 60% of a job can be made parallel. We use 3 processors:
1.67x faster than with 1 worker

19. Amdahl’s Law Always have to do 40% of the work in serial
With infinite workers:
𝑆𝑝𝑒𝑒𝑑𝑢𝑝 𝑁 = 1 1− ∞

20. Amdahl’s Law Always have to do 40% of the work in serial
With infinite workers: Only 2.5x faster!
𝑆𝑝𝑒𝑒𝑑𝑢𝑝 𝑁 = 1 1− ∞ = = 2.5

21. Limits
Max speedup limited by parallel portion of code:

22. Speedup Issues : Overhead
Even assuming no sequential portion, there’s…
Time to think how to divide the problem up
Time to hand out small “work units” to workers
All workers may not work equally fast
Some workers may fail
There may be contention for shared resources
Workers could overwriting each others’ answers
You may have to wait until the last worker returns to proceed (the slowest / weakest link problem)
There’s time to merge the results back together

23. Why Parallelism? We have no choice! Parallel processing takes new
Multicore processors are a plan B
Parallel processing takes new
Computer Architectures
Algorithms
Programming Tools