1. Lappeenranta University of Technology / JP CT30A7001 Concurrent and Parallel Computing Introduction to concurrent and parallel computing

2. Lappeenranta University of Technology / JP Definitions Concurrent computing –Concurrent computing is the concurrent (simultaneous) execution of multiple interacting computational tasks. Concurrent computing is related to parallel computing, but focuses more on the interactions between tasks. Parallel computing –Parallel computing is a form of computation in which many instructions are carried out simultaneously. Distributed computing –In distributed computing a program is split up into parts that run simultaneously on multiple computers communicating over a network. Distributed computing is a form of parallel computing, but parallel computing is most commonly used to describe program parts running simultaneously on multiple processors in the same computer. Grid computing –Grid computing is a form of distributed computing whereby a "super and virtual computer" is composed of a cluster of networked, loosely-coupled computers, acting in concert to perform very large tasks.

3. Lappeenranta University of Technology / JP Motivation The ever increasing need of computing power has traditionally been fulfilled by producing new more powerful processors => physical limits restrict the growth ( Clock rates have increased from 40 MHz (1988) to over 2 GHz (2002); Multiple instructions/cycle, multiple processors)

4. Lappeenranta University of Technology / JP Performance – Cost ratio Cost Performance 1960s 1990s 1980s 1970s

5. Lappeenranta University of Technology / JP Can’t you go any faster ? More power can be achieved by using multiple computers (parallel computing) => allows us to break the limits set for the single processor

6. Lappeenranta University of Technology / JP Trends ? Faster processors will evolve !! –Moore’s law (statement) –Development of parallel applications take time ? However –Memory access times lag behind (10% growth) –The data transfer speed lags behind the processing capacity –Software lags behind the hardware

7. Lappeenranta University of Technology / JP Moore’s law

8. Lappeenranta University of Technology / JP Parallel computing ? Non-”von Neumann” type of computing Some problem is solved at the same time in two or more locations (processors …) –Example: library workers –Problem need to have some inherent parallelism –Problem is solved in some parallel architecture –Problem is solved by some algorithm

9. Lappeenranta University of Technology / JP Library workers example 1 or more workers return books into shelves Aisles are divided according to the alphabet –A-E, F-J, K-O, P-T, U-Z How to divide work among workers ?

10. Lappeenranta University of Technology / JP 1 Worker Algorithm Work

11. Lappeenranta University of Technology / JP 2 Workers

12. Lappeenranta University of Technology / JP 2 Workers

13. Lappeenranta University of Technology / JP 2 Workers A-E F-J K-O P-T U-Z

14. Lappeenranta University of Technology / JP Things to consider The amount of work –When do we need several workers ? The number of workers –How many workers is enough ? Internal algorithm –How do we divide the work ? External algorithm (communication) –How is the work is synchronized

15. Lappeenranta University of Technology / JP Utilization

16. Lappeenranta University of Technology / JP Inherent parallelism Parallel computing can only be applied to those problems that have some inherent parallelism Parallelism is an inherent property that cannot be increased ! –Some problems are naturally sequential –One can affect how well the parallelism is used

17. Lappeenranta University of Technology / JP Benefits of parallel computing Many problems are naturally parallel although they are modeled sequentially –Interrupts, polling => loss of information There are problems that cannot be solved sequentially Performance may increase linearly (some times even more) In future natural parallelism is commonly utilized –There are no sequential computers (inherent parallelism)

18. Lappeenranta University of Technology / JP Parallel computing goals Different types of goals: –Reduce the execution time Example: weather forecasts –Solve bigger problems (more accurate models) Example: Mobile network simulations –Run multiple applications at the same time Multithreading –Natural approach

19. Lappeenranta University of Technology / JP Example: Weather forecast Compute weather forecast for 3000*3000 miles area up to 11 miles high Divide the model into parts of size 0.1*0.1*0.1 miles => 10 11 parts Compute weather forecast for two days (48 hours) Computing one part takes ~100 operations and parts are computed ~100 times within these 48 hours => 10 15 operations to compute If we have a single 1000 MIPS workstation computations take ~280 hours !!! If we use 1000 workstations capable of 100 MIPS the computation takes only ~3 hours

20. Lappeenranta University of Technology / JP Applicability ? Parallel computing can be used for solving all those problems that contain some parallel properties Time consuming task ! –Performance –Portability –Scalability

21. Lappeenranta University of Technology / JP Application areas Technical applications –floating point operations, computing grids Business applications –decision support systems, simulations,... Network applications –multimedia, video-on-demand,...

22. Lappeenranta University of Technology / JP

23. Parallel architectures Problems are computed in parallel in parallel architectures There exits several different approaches: –Massively parallel processors (MPP) –Symmetrical multiprocessor (SMP) –Vector -processors –Cluster machines

24. Lappeenranta University of Technology / JP Examples Cray-1 Avalon Cluster - 140

25. Lappeenranta University of Technology / JP RankManufacturer Computer/ProcsR max R peak Installation Site Country/Year 1NEC Earth-Simulator/ 5120 35860.00 40960.00 Japan/2002 2Hewlett-Packard ASCI Q - AlphaServer SC ES45/1.25 GHz/ 4096 7727.00 10240.00 USA/2002 3Hewlett-Packard ASCI Q - AlphaServer SC ES45/1.25 GHz/ 4096 7727.00 10240.00 USA/2002 4IBM ASCI White, SP Power3 375 MHz/ 8192 7226.00 12288.00 USA/2000 5Linux NetworX MCR Linux Cluster Xeon 2.4 GHz - Quadrics/ 2304 5694.00 11060.00 USA/2002 6Hewlett-Packard AlphaServer SC ES45/1 GHz/ 3016 4463.00 6032.00 USA/2001 7Hewlett-Packard AlphaServer SC ES45/1 GHz/ 2560 3980.00 5120.00 France/2001 8HPTi Aspen Systems, Dual Xeon 2.2 GHz - Myrinet2000/ 1536 3337.00 6758.00 USA/2002 9IBM pSeries 690 Turbo 1.3GHz/ 1280 3241.00 6656.00 UK/2002 10IBM pSeries 690 Turbo 1.3GHz/ 1216 3164.00 6323.00 USA/2002 TOP500 - 2003

26. Lappeenranta University of Technology / JP Top 500 - 2006

27. TOP 500 - 2010 Lappeenranta University of Technology / JP

28. No 54

29. Lappeenranta University of Technology / JP

33. Hyperthreading within processor Use of threads within workstation Message passing inside cluster GRID Number of possible users increases User interaction increases (requires knowledge) Geographic size increases

34. Lappeenranta University of Technology / JP Algorithms Selection of the algorithm is important as different algorithms have different suitability The fastest sequential algorithm may not be the best algorithm to be parallelized ! The selection of algorithm may depend on the selected parallel architecture

35. Lappeenranta University of Technology / JP Example: Sorting a sequence Traditional approach: O(nlogn) time Simple Merge-Sort algorithm: O(logn loglogn) time Pipelined sorting algorithm: O(logn) time

36. Lappeenranta University of Technology / JP

37. Note ! Parallel computing does NOT help if the bottleneck of the system is in memory, in disk system or in network ! Parallel computing may produce surprises –resources are competed => winner is not known beforehand => non-deterministic behavior of the program => deadlock ? (mutual exclusion)

38. Lappeenranta University of Technology / JP Aspects slowing the future Not all problems can be solved in parallel –Example: Digging a well vs. a dike Parallel programming requires a new way of thinking (compare OO-programming) Sequential algorithm may not be suitable for parallel execution Partitioning and load balancing are not an easy task Tools are to be developed at the moment

39. Lappeenranta University of Technology / JP Tools for parallel computing Threads –http://www.sun.com/software/Products/Developer- products/threads/http://www.sun.com/software/Products/Developer- products/threads/ PVM (Parallel Virtual Machine) –http://www.epm.ornl.gov/pvm/http://www.epm.ornl.gov/pvm/ MPI (Message Passing Interface) –http://www.epm.ornl.gov/~walker/mpi/http://www.epm.ornl.gov/~walker/mpi/ –http://www.netlib.org/utk/papers/mpi-book/mpi-book.htmlhttp://www.netlib.org/utk/papers/mpi-book/mpi-book.html OpenMP