1. Parallel Programming in C with MPI and OpenMP Michael J. Quinn

2. Chapter 1 Motivation and History

3. . Outline Motivation Motivation Modern scientific method Modern scientific method Evolution of supercomputing Evolution of supercomputing Modern parallel computers Modern parallel computers Seeking concurrency Seeking concurrency Programming parallel computers Programming parallel computers

4. . What is Parallel and Distributed computing?  Solving a single problem faster using multiple CPUs  E.g. Matrix Multiplication C = A X B  Parallel = Shared Memory among all CPUs  Distributed = Local Memory/CPU  Common Issues: Partition, Synchronization, Dependencies, load balancing

5. . Why Parallel and Distributed Computing? Grand Challenge Problems Grand Challenge Problems  Weather Forecasting; Global Warming  Materials Design – Superconducting material at room temperature; nano- devices; spaceships.  Organ Modeling; Drug Discovery

6. . Why Parallel and Distributed Computing? Physical Limitations of Circuits Physical Limitations of Circuits  Heat and light effect  Superconducting material to counter heat effect  Speed of light effect – no solution!

7. . Microprocessor Revolution Micros Minis Mainframes Speed (log scale) Time Supercomputers

8. . VLSI – Effect of Integration VLSI – Effect of Integration  1 M transistor enough for full functionality - Dec’s Alpha (90’s)  Rest must go into multiple CPUs/chip Cost – Multitudes of average CPUs give better FLPOS/$ compared to traditional supercomputers Cost – Multitudes of average CPUs give better FLPOS/$ compared to traditional supercomputers Why Parallel and Distributed Computing?

9. . Modern Parallel Computers Caltech’s Cosmic Cube (Seitz and Fox) Caltech’s Cosmic Cube (Seitz and Fox) Commercial copy-cats Commercial copy-cats  nCUBE Corporation (512 CPUs)  Intel’s Supercomputer Systems  iPSC1, iPSC2, Intel Paragon (512 CPUs)  Lots more Thinking Machines Corporation Thinking Machines Corporation  CM2 (65K 4-bit CPUs) – 12-dimensional hypercube - SIMD  CM5 – fat-tree interconnect - MIMD   Roadrunner - Los Alamos NL 116,640 cores 12K IBM cell

10. . Everyday Reasons Everyday Reasons  Available local networked workstations and Grid resources should be utilized  Solve compute-intensive problems faster  Make infeasible problems feasible  Reduce design time  Leverage of large combined memory  Solve larger problems in same amount of time  Improve answer’s precision  Reduce design time  Gain competitive advantage  Exploit commodity multi-core and GPU chips Why Parallel and Distributed Computing?

11. . Why MPI/PVM? MPI = “Message Passing Interface” MPI = “Message Passing Interface” PVM = “Parallel Virtual Machine” PVM = “Parallel Virtual Machine” Standard specification for message-passing libraries Standard specification for message-passing libraries Libraries available on virtually all parallel computers Libraries available on virtually all parallel computers Free libraries also available for networks of workstations, commodity clusters, Linux, Unix, and Windows platforms Free libraries also available for networks of workstations, commodity clusters, Linux, Unix, and Windows platforms Can program in C, C++, and Fortran Can program in C, C++, and Fortran

12. . Why Shared Memory programming? Easier conceptual environment Easier conceptual environment Programmers typically familiar with concurrent threads and processes sharing address space Programmers typically familiar with concurrent threads and processes sharing address space CPUs within multi-core chips share memory CPUs within multi-core chips share memory OpenMP an application programming interface (API) for shared-memory systems OpenMP an application programming interface (API) for shared-memory systems  Supports higher performance parallel programming of symmetrical multiprocessors

13. . Classical Science Nature Observation Theory Physical Experimentation

14. . Modern Scientific Method Nature Observation Theory Physical Experimentation Numerical Simulation

15. . Evolution of Supercomputing World War II World War II  Hand-computed artillery tables  Need to speed computations  ENIAC Cold War Cold War  Nuclear weapon design  Intelligence gathering  Code-breaking

16. . Advanced Strategic Computing Initiative U.S. nuclear policy changes U.S. nuclear policy changes  Moratorium on testing  Production of new weapons halted Numerical simulations needed to maintain existing stockpile Numerical simulations needed to maintain existing stockpile Five supercomputers costing up to $100 million each Five supercomputers costing up to $100 million each

17. . ASCI White (10 teraops/sec) Mega flops = 10^6 flops = 2^20 Giga = 10^9 = billion = 2^30 Tera = 10^12 = trillion = 2^40 Peta = 10^15 = quadrillion = 2^50 Exa = 10^18 = quintillion = 2^60

18. . Eniac (350 op/s) 1946 - (U.S. Army photo)

19. . Supercomputer Fastest General-purpose computer Fastest General-purpose computer Solves individual problems at high speeds, compared with contemporary systems Solves individual problems at high speeds, compared with contemporary systems Typically costs $10 million or more Typically costs $10 million or more Traditionally found in government labs Traditionally found in government labs

20. . Commercial Supercomputing Started in capital-intensive industries Started in capital-intensive industries  Petroleum exploration  Automobile manufacturing Other companies followed suit Other companies followed suit  Pharmaceutical design  Consumer products  Financial Transactions

21. . 60 Years of Speed Increases ENIAC 350 flops 1946 Today - 2008  1 Peta flops  Roadrunner - Los Alamos NL 116,640 cores 12K IBM cell

22. . CPUs Millions of Times Faster Faster clock speeds Faster clock speeds Greater system concurrency Greater system concurrency  Multiple functional units  Concurrent instruction execution  Speculative instruction execution Systems 1 Trillion Times Faster Systems 1 Trillion Times Faster  Processors are millions times faster  Combine hundred thousands of processors

23. . Modern Parallel Computers Caltech’s Cosmic Cube (Seitz and Fox) Caltech’s Cosmic Cube (Seitz and Fox) Commercial copy-cats Commercial copy-cats  nCUBE Corporation (512 CPUs)  Intel’s Supercomputer Systems  iPSC1, iPSC2, Intel Paragon (512 CPUs)  Lots more Thinking Machines Corporation Thinking Machines Corporation  CM2 (65K 4-bit CPUs) – 12-dimensional hypercube - SIMD  CM5 – fat-tree interconnect - MIMD

24. . Copy-cat Strategy Microprocessor Microprocessor  1% speed of supercomputer  0.1% cost of supercomputer Parallel computer = 1000 microprocessors Parallel computer = 1000 microprocessors  10 x speed of traditional supercomputer  Same cost as supercomputer

25. . Why Didn’t Everybody Buy One? Supercomputer   CPUs Supercomputer   CPUs  Computation rate  throughput (#jobs/time)  Slow Interconnect  Inadequate I/O  Customized Compute and Communication processors meant inertia in adopting the fastest commodity chip with least cost and effort  Focus on high end computation meant no sales volume to reduce cost Software Software  Inadequate operating systems  Inadequate programming environments

26. . After the “Shake Out” IBM – SP1 and SP2, Blue Gene L/P IBM – SP1 and SP2, Blue Gene L/P Hewlett-Packard – Tandem Hewlett-Packard – Tandem Silicon Graphics (Cray) – Origin Silicon Graphics (Cray) – Origin Sun Microsystems - Starfire Sun Microsystems - Starfire

27. . Commercial Parallel Systems Relatively costly per processor Relatively costly per processor Primitive programming environments Primitive programming environments Focus on commercial sales Focus on commercial sales Scientists looked for alternative Scientists looked for alternative

28. . Beowulf Cluster Concept NASA (Sterling and Becker) NASA (Sterling and Becker) Commodity processors Commodity processors Commodity interconnect Commodity interconnect Linux operating system Linux operating system MPI/PVM library MPI/PVM library High performance/$ for certain applications High performance/$ for certain applications

29. . Seeking Concurrency Data dependence graphs Data dependence graphs Data parallelism Data parallelism Functional parallelism Functional parallelism Pipelining Pipelining

30. . Data Dependence Graph Directed graph Directed graph Vertices = tasks Vertices = tasks Edges = dependencies Edges = dependencies

31. . Data Parallelism Independent tasks apply same operation to different elements of a data set Independent tasks apply same operation to different elements of a data set Okay to perform operations concurrently Okay to perform operations concurrently Speedup: potentially p-fold, p=#processors Speedup: potentially p-fold, p=#processors for i  0 to 99 do a[i]  b[i] + c[i] endfor

32. . Functional Parallelism Independent tasks apply different operations to different data elements Independent tasks apply different operations to different data elements First and second statements First and second statements Third and fourth statements Third and fourth statements Speedup: Limited by amount of concurrent sub- tasks Speedup: Limited by amount of concurrent sub- tasks a  2 b  3 m  (a + b) / 2 s  (a 2 + b 2 ) / 2 v  s - m 2

33. . Pipelining Divide a process into stages Divide a process into stages Produce several items simultaneously Produce several items simultaneously Speedup: Limited by amount of concurrent sub- tasks = #of stages in the pipeline Speedup: Limited by amount of concurrent sub- tasks = #of stages in the pipeline

34. . Programming Parallel Computers Extend compilers: translate sequential programs into parallel programs Extend compilers: translate sequential programs into parallel programs Extend languages: add parallel operations Extend languages: add parallel operations Add parallel language layer on top of sequential language Add parallel language layer on top of sequential language Define totally new parallel language and compiler system Define totally new parallel language and compiler system

35. . Strategy 1: Extend Compilers Parallelizing compiler Parallelizing compiler  Detect parallelism in sequential program  Produce parallel executable program Focus on making Fortran programs parallel Focus on making Fortran programs parallel

36. . Extend Compilers (cont.) Advantages Advantages  Can leverage millions of lines of existing serial programs  Saves time and labor  Requires no retraining of programmers  Sequential programming easier than parallel programming

37. . Extend Compilers (cont.) Disadvantages Disadvantages  Parallelism may be irretrievably lost when programs written in sequential languages  Parallelizing technology works mostly for easy codes with loops, etc.  Performance of parallelizing compilers on broad range of applications still up in air

38. . Extend Language Add functions to a sequential language Add functions to a sequential language  Create and terminate processes  Synchronize processes  Allow processes to communicate  E.g., MPI, PVM, Process/thread, OpenMP

39. . Extend Language (cont.) Advantages Advantages  Easiest, quickest, and least expensive  Allows existing compiler technology to be leveraged  New libraries can be ready soon after new parallel computers are available

40. . Extend Language (cont.) Disadvantages Disadvantages  Lack of compiler support to catch errors  Easy to write programs that are difficult to debug

41. . Add a Parallel Programming Layer Lower layer Lower layer  Core of computation  Process manipulates its portion of data to produce its portion of result (persistent object like) Upper layer Upper layer  Creation and synchronization of processes  Partitioning of data among processes A few research prototypes have been built based on these principles – Linda, iC2Mpi platform (Prasad, et al. IPDPS-07 workshops), SyD Middleware – System on Devices (Prasad, et al., MW-04) A few research prototypes have been built based on these principles – Linda, iC2Mpi platform (Prasad, et al. IPDPS-07 workshops), SyD Middleware – System on Devices (Prasad, et al., MW-04)

42. . Create a Parallel Language Develop a parallel language “from scratch” Develop a parallel language “from scratch”  Occam is an example Add parallel constructs to an existing language Add parallel constructs to an existing language  Fortran 90  High Performance Fortran  C*

43. . New Parallel Languages (cont.) Advantages Advantages  Allows programmer to communicate parallelism to compiler  Improves probability that executable will achieve high performance Disadvantages Disadvantages  Requires development of new compilers  New languages may not become standards  Programmer resistance

44. . Current Status Low-level approach is most popular Low-level approach is most popular  Augment existing language with low-level parallel constructs  MPI, PVM, threads/process-based concurrency and OpenMP are examples Advantages of low-level approach Advantages of low-level approach  Efficiency  Portability Disadvantage: More difficult to program and debug Disadvantage: More difficult to program and debug

45. . Summary High performance computing High performance computing  U.S. government  Capital-intensive industries  Many companies and research labs Parallel computers Parallel computers  Commercial systems  Commodity-based systems

46. . Summary – contd. Power of CPUs keeps growing exponentially Power of CPUs keeps growing exponentially Parallel programming environments changing very slowly Parallel programming environments changing very slowly Two standards have emerged Two standards have emerged  MPI/PVM library, for processes that do not share memory  Process/thread based concurrency and OpenMP directives for processes that do share memory