1. Parallel Processing Sharing the load

2. Inside a Processor Chip in Package Circuits Primarily Crystalline Silicon 1 mm – 25 mm on a side 100 million to billions of transistors – current “feature size” (process) ~ 22 nanometers Package provides: – communication with motherboard – heat dissipation

3. Moore's Law Number of transistors in same area doubles every 2 years Net effects: Processing power doubles approximately every 18 months

4. Adapted from UC Berkeley "The Beauty and Joy of Computing" Exponential Growth Doubling is exponential growth Year 01.534.567.5910.512 Speed1248163264128256

5. Adapted from UC Berkeley "The Beauty and Joy of Computing" Moore's Law Gordon Moore Intel Cofounder

6. Adapted from UC Berkeley "The Beauty and Joy of Computing" Moore's Law If Moore's Law were applicable to the airline industry, a flight from New York to Paris in 1978 that cost $900 and took seven hours, would now cost about $0.01 and take less than one second.

7. Adapted from UC Berkeley "The Beauty and Joy of Computing" Power Density Prediction circa 2000 Core 2

8. MultiCore Multicore : Multiple processing cores on one chip – Each core can run a different program

9. Adapted from UC Berkeley "The Beauty and Joy of Computing" Going Multi-core Helps Energy Efficiency Speed takes power, Power = heat – Can run at 80% speed with 50% power

10. Adapted from UC Berkeley "The Beauty and Joy of Computing" Moore's Law Related Curves

11. Adapted from UC Berkeley "The Beauty and Joy of Computing" Moore's Law Related Curves

12. Adapted from UC Berkeley "The Beauty and Joy of Computing" Issues Not every part of a problem scales well – Parallel : can run at same time – Serial : must run one at a time in order

13. Adapted from UC Berkeley "The Beauty and Joy of Computing" 5 workers can do parallel portion in 1/5 th the time Can't affect serial part Speedup Issues Time Number of Cores Parallel portion Serial portion 1 5

14. Adapted from UC Berkeley "The Beauty and Joy of Computing" Speedup Issues Time Number of Cores Parallel portion Serial portion 123 45 Increasing workers provide diminishing returns

15. Adapted from UC Berkeley "The Beauty and Joy of Computing" Amdahl’s Law Amdahl’s law : Predicts how many times faster N workers can do a task in which P portion is parallel

16. Adapted from UC Berkeley "The Beauty and Joy of Computing" Amdahl’s Law 60% of a job can be made parallel. We use 2 processors: 1.43x faster with 2 than 1

17. Adapted from UC Berkeley "The Beauty and Joy of Computing" Amdahl’s Law 60% of a job can be made parallel. We use 3 processors: 1.67x faster than with 1 worker

18. Adapted from UC Berkeley "The Beauty and Joy of Computing" Amdahl’s Law Always have to do 40% of the work in serial With infinite workers: Only 2.5x faster!

19. Adapted from UC Berkeley "The Beauty and Joy of Computing" Limits Max speedup limited by parallel portion of code:

20. Adapted from UC Berkeley "The Beauty and Joy of Computing" Speedup Issues : Overhead Even assuming no sequential portion, there’s… – Time to think how to divide the problem up – Time to hand out small “work units” to workers – All workers may not work equally fast – Some workers may fail – There may be contention for shared resources – Workers could overwriting each others’ answers – You may have to wait until the last worker returns to proceed (the slowest / weakest link problem) – There’s time to put the data back together in a way that looks as if it were done by one

21. Adapted from UC Berkeley "The Beauty and Joy of Computing" Concurrency Concurrency : two things happening at the same time Many things don't work well concurrently – Printers – Shared memory

22. Adapted from UC Berkeley "The Beauty and Joy of Computing" No Synchronization Race Condition : unpredictable result based on timing of concurrent operations

23. Adapted from UC Berkeley "The Beauty and Joy of Computing" X starts as 5, four possible answers: No Synchronization Case 1Case 2Case 3Case 4 A runs, x = 15 B runs, x =16 B runs, x = 6 A runs, x = 16 A gets x (5) B gets x (5) A adds 10 (has 15) B adds 1 (has 6) A stores x = 15 B stores x = 6 A gets x (5) B gets x (5) A adds 10 (has 15) B adds 1 (has 6) B stores x = 6 A stores x = 15

24. Adapted from UC Berkeley "The Beauty and Joy of Computing" Locks Can prevent concurrency problems with locks:

25. Adapted from UC Berkeley "The Beauty and Joy of Computing" Deadlock But if we have… – Mutual exclusion : can't share resources – Hold and wait : you can reserve one resource while waiting on another – No preemption : can't remove a resource from a process's control Can have deadlock…

26. Adapted from UC Berkeley "The Beauty and Joy of Computing" Deadlock Workers A and B both want to use locked resources X and Y:

27. Adapted from UC Berkeley "The Beauty and Joy of Computing" Breaking Deadlock Must remove one condition – Mutual Exclusion Find a way to share – Hold and Wait If you wait you must give up other resources – No preemption Take back a resource someone has claimed

28. Adapted from UC Berkeley "The Beauty and Joy of Computing" Why Parallelism? We have no choice! – Multicore processors are a plan B, not a triumph for parallelism Parallel processing takes new – Architectures – Algorithms – Structures – Languages