1. Yulia Newton CS 147, Fall 2009 SJSU

2. What is it? “Parallel processing is the ability of an entity to carry out multiple operations or tasks simultaneously. The term is used in the contexts of both human cognition and machine computation.” - Wikipedia

3. Humans Human body is a parallel processing architecture Human body is the most complex machine known to man We perform parallel processing all the time

4. Why? We want to do MORE We want to do it FASTER

5. Limitations of Single Processor Physical Heat Electromagnetic interference Economic Cost

6. Parallel Computing Divide bigger tasks into smaller tasks Reminds you Divide and Conquer concept? Perform calculations of those tasks simultaneously (in parallel)

7. With Single CPU Parallel processing is possible with single-CPU, single- core computers Requires very sophisticated software called distributed processing software Most parallel computing is done with multiple CPU’s

8. Speedup from parallelization More processors = faster computing? Speed-up from parallelization is not linear Can do single task in a unit of time Can do twice as many tasks in same the amount of time Can do four times as many tasks in the same amount of time

9. Speedup Definition: Speedup is how much a parallel algorithm is faster than a corresponding sequential algorithm.

10. Gene Myron Amdahl Norwegian American computer architect and hi-tech entrepreneur

11. Amdahl's Law Formulated in 1960s Gives potential speed-up on parallel computing system Small portion of components/elements that cannot be parallelized will limit speed-up possible from parallelizable components/elements.

12. Amdahl's Law (cont’d) where : S is the overall speed-up of the system P is the fraction that is parallelizable

13. Amdahl's Law Alternative Form where f – fraction of the program that is enhanced s – speedup of the enhanced portion

14. Amdahl's Law (cont’d)

15. Program Amdahl's Law Example Program has four parts Code 1 Loop 1 Loop 2 Code 2 Part 1 = 11% Part 2 = 18% Part 3 = 23% Part 4 = 48%

16. Amdahl's Law Example (cont’d) Let’s say: P1 is not sped up, so S1 = 1 or 100% P2 is sped up 5×, so S2 = 500% P3 is sped up 20×, so S3 = 2000% P4 is sped up 1.6×, so S4 = 160%

17. Amdahl's Law Example (cont’d) Using P1/S1 + P2/S2 + P3/S3 + P4/S4:

18. Amdahl's Law Example (cont’d) 0.4575 is a little less than ½ of the original running time, which is 1 Overall speed boost is 1 / 0.4575 = 2.186 or a little more than double the original speed Notice how the 20× and 5× speedup don't have much effect on the overall speed boost and running time when 11% is not sped up, and 48% is sped up by 1.6×

19. Amdahl's Law Limitations Based on fixed work load or fixed problem size (strong or hard scaling) Implies that machine size is irrelevant

20. John L. Gustafson American computer scientist and businessman

21. Gustafson's Law Formulated in 1988 Closely related to Amdahl’s Law Addresses the shortcomings of Amdahl's law, which cannot scale to match availability of computing power as the machine size increases

22. Gustafson's Law (cont’d) where P is the number of processors, S is the speedup, and α the non-parallelizable part of the process

23. Gustafson's Law (cont’d) Proposes a fixed time concept which leads to scaled speed up for larger problem sizes Uses weak or soft scaling

24. Gustafson's Law Limitations Some problems do not have fundamentally larger datasets Example: processing one data point per world citizen gets larger at only a few percent per year Nonlinear algorithms may make it hard to take advantage of parallelism "exposed" by Gustafson's law

25. Parallel Architectures Superscalar VLIW Vector Processors Interconnection Networks Shared Memory Multiprocessors Distributed Computing Dataflow Computing Neural Networks Systolic Arrays

26. Superscalar Architecture Implements instruction-level parallelism (multiple instructions are executed simultaneously in each cycle) Net effect is the same as pipelining Additional hardware is required Contains specialized instruction fetch unit (retrieves multiple instructions simultaneously from memory) Relies on both hardware and compiler

27. Superscalar Architecture Analogous to adding a lane to a highway – more physical resources are required but in the end more cars can pass through

28. Superscalar Architecture Examples: Pentium x86 processors Intel i960CA AMD 29000-series

29. VLIW Architecture Stands for Very Long Instruction Word Similar to Superscalar architecture Relies entirely on compiler Pack independent instructions into one long instruction Bigger size of compiled code

30. VLIW Architecture Examples: Intel’s Itanium, IA-64 Floating Point Systems' FPS164

31. Vector Processors Supercomputers are vector processor systems Specialized, heavily pipelined processors Efficient operations on entire vectors and matrices

32. Vector Processors Heavily pipelined so that arithmetic operations can be overlapped

33. Vector Processors (example) Instructions on a traditional processor: For I = 0 to VectorLength V3[i] = V1[i] + V2[i] Instructions on a vector processor: LDV V1, R1 LDV V2, R2 ADDV R3, R1, R2 STV R3, V3

34. Vector Processors Applications: Weather forecasting Medical diagnoses systems Image processing

35. Vector Processors Examples: Cray series of cupercomputers Xtrillion 3.0

36. Interconnection Networks MIMD (Multiple instruction stream, multiple data stream) type architecture Each processor has its own memory Processors are allowed to access other processors’ memory via network

37. Interconnection Networks Can be: Dynamic – allow paths between two components to change from one communication to another Static – do not allow a change of path between communications

38. Interconnection Networks Can be: Non-blocking – allow new connections in the presence of another simultaneous connections Blocking – do not allow simultaneous connections

39. Interconnection Networks Sample interconnection networks:

40. Shared Memory Architecture MIMD (Multiple instruction stream, multiple data stream) type architecture Single memory pool accessed by all processors Can be categorized by how the memory access is performed: UMA (Uniform Memory Access) – all processors have equal access to entire memory NUMA (Non-Uniform Memory Access) – each processor has access to its own piece of memory

41. Shared Memory Architecture Diagram of a Shared Memory system:

42. Distributed Computing Very loosely coupled multicomputer system, connected by buses or through a network Main advantage is cost Main disadvantage is slow communication due to the physical distance between computing units

43. Distributed Computing Example: SETI (Search for Extra Terrestrial Intelligence) group at UC Berkley analyzes data from radio-telescopes To help this project PC users can install SETI screen saver on their home computer Data will be analyzed during the processor’s idle time Half a million years of CPU time accumulated in about 18 months!

44. Data Flow Computing Von Neumann machines exhibit sequential control flow; data and instructions are segregated Dataflow computing – control of the program is directly tied to the data Order of the instructions does not matter Each instruction is considered a separate process Tokens are executed rather than instructions

45. Data Flow Computing Data Flow Graph for y=(c+3)*f

46. Data Flow Computing Data flow system diagram:

47. Neural Networks Good for massively parallel applications, fault tolerance and adapting to changing circumstances Based on parallel architecture of human brain Composed of large number of simple processing elements Trainable

48. Neural Networks Model human brain Perceptron is the simplest example

49. Neural Networks Applications: Quality control Financial and economic forecasting Oil and gas exploration Speech and pattern recognition Health care cost reduction Bankruptcy prediction

50. Systolic Arrays Networks of processing elements that rhythmically compute data by circulating it through the system Variations of SIMD architecture

51. Systolic Arrays Chain of processors in a systolic array system:

52. To Recap We completed overview of the following parallel processing architectures: Superscalar VLIW Vector Processors Interconnection Networks Shared Memory Multiprocessors Distributed Computing Dataflow Computing Neural Networks Systolic Arrays

53. References and Sources Null, Linda and Julia Lobur. “Computer Organization and Architecture” Null, Linda and Julia Lobur. “Computer Organization and Architecture” http://en.wikipedia.org http://www.cs.berkeley.edu/~culler/cs258-s99 http://www.cs.iastate.edu/~prabhu http://www.answers.com http://www.gigaflop.demon.co.uk

54. Questions? Thank you!