PARALLEL COMPUTING Submitted By : P. Nagalakshmi Submitted To: Dr. R. K. Rathy HOD- CSE

Out line of the Seminar Parallel Computing Architecture Taxonomy Parallel Computer Memory Architectures Parallel Programming Models Steps for Creating a Parallel Program Design and Performance Considerations Examples

Overview of Parallel Computing Parallel computing is the simultaneous execution of the same task on multiple processors in order to obtain faster results. In the simplest sense, parallel computing is the simultaneous use of multiple compute resources to solve a computational problem. The compute resources can include: A single computer with multiple processors; An arbitrary number of computers connected by a network; A combination of both

Parallel Computing A problem is broken into discrete parts that can be solved concurrently Each part is further broken down to a series of instructions Instructions from each part execute simultaneously on different CPUs

Why Use Parallel Computing? Save time Solve larger problems Provide concurrency (do multiple things at the same time) It is expensive making single processor faster

Von Neumann Architecture Memory is used to store both program and data instructions Program instructions are coded data which tell the computer to do something Data is simply information to be used by the program A central processing unit (CPU) gets instructions and/or data from memory, decodes the instructions and then sequentially performs them.

Architecture Taxonomy A serial (non-parallel) computer Single instruction Single data A type of parallel computer Single instruction Multiple data Architecture Taxonomy A single data stream is fed into multiple processing units. Each processing unit operates on the data independently S I S D Single Instruction, Single Data S I M D Single Instruction, Multiple Data M I S D Multiple Instruction, Single Data M I M D Multiple Instruction, Multiple Data The most common type of parallel computer Multiple Instruction Multiple Data

Memory Architectures Shared Memory Distributed Memory Hybrid Distributed-Shared Memory

The ability for all processors to access all memory as global address space. Multiple processors can operate independently but share the same memory resources. Changes in a memory location effected by one processor are visible to all other processors. Shared memory machines can be divided into two main classes based upon memory access times: UMA and NUMA Shared Memory

Distributed Memory Require a communication network to connect inter-processor memory There is no concept of global address space across all processors. Processors operates independently. Hence, the concept of cache coherency does not apply.

Hybrid Distributed-Shared Memory The largest and fastest computers in the world today employ both shared and distributed memory architectures.

Parallel Programming Models Threads Message Passing Data Parallel Other Models

Threads Model A single process can have multiple, concurrent execution paths.

Message Passing Model Tasks exchange data through communications by sending and receiving messages. Data transfer usually requires cooperative operations to be performed by each process. For example, a send operation must have a matching receive operation.

Data Parallel Model Most of the parallel work focuses on performing operations on a data set. The data set is typically organized into a common structure, such as an array or cube. A set of tasks work collectively on the same data structure, however, each task works on a different partition of the same data structure.

Other Models Hybrid SPMD Combination of two or more parallel programming model. SPMD High level programming model. Single program can be executed by all tasks. All task may use different data.

Other Models MPMD High level programming model. Each task can execute same or different program. All task may use different data.

Steps for Creating a Parallel Program If you are starting with an existing serial program, debug the serial code completely Identify the parts of the program that can be executed concurrently Decompose the program Code development Compile, Test, Debug Optimization

Design and Performance Considerations Amdahl's Law Amdahl's Law states that potential program speedup is defined by the fraction of code (P) which can be parallelized speedup = 1 / 1 - P 1 speedup = ------------ P + S --- N where P = parallel fraction, N = number of processors and S = serial fraction.

Design and Performance Considerations (contd) Load Balancing Load balancing refers to the distribution of tasks in such a way as to insure the most time efficient parallel execution If tasks are not distributed in a balanced way, you may end up waiting for one task to complete while other tasks are idle Performance can be increased if work can be more evenly distributed

Design and Performance Considerations (contd) Granularity The ratio between computation and communication is known as granularity. Fine-grain parallelism All tasks execute a small number of instructions between communication cycles. Low computation to communication ratio. Facilitates load balancing. If granularity is too fine it is possible that the overhead required for communications and synchronization between tasks takes longer than the computation.

Design and Performance Considerations (contd) Coarse-grain parallelism Typified by long computations consisting of large numbers of instructions between communication synchronization points High computation to communication ratio Implies more opportunity for performance increase Harder to load balance efficiently

Design and Performance Considerations (contd) Data Dependency A data dependency exists when there is multiple use of the same storage location Example 1: DO 500 J = MYSTART,MYEND A(J) = A(J-1) * 2.0 500 CONTINUE This code has a data dependency. Must have computed value for A(J-1) before we can calculate A(J). If Task 2 has A(J) and Task 1 has A(J-1), the value of A(J) is dependent on

Design and Performance Considerations (contd) Deadlock A condition where two or more processes are waiting for an event or communication from one of the other processes. TASK1 TASK2 ------------------ ------------------ X = 4 Y = 8 SOURCE = TASK2 SOURCE = TASK1 RECEIVE (SOURCE,Y) RECEIVE (SOURCE,X) DEST = TASK2 DEST = TASK1 SEND (DEST, X) SEND (DEST, Y) Z = X + Y Z = X + Y

Design and Performance Considerations (contd) Debugging Debugging parallel programs is significantly more of a challenge than debugging serial programs Parallel debuggers are beginning to become available, but much work remains to be done Use a modular approach to program development Pay as close attention to communication details as to computation details

Design and Performance Considerations (contd) Performance Monitoring and Analysis As with debugging, monitoring and analyzing parallel program execution is significantly more of a challenge than for serial programs A number of parallel tools for execution monitoring and program analysis are available Some are quite useful; some are cross-platform also Work remains to be done, particularly in the area of scalability.

Places where Parallel computing used Weather and ocean patterns Automobile assembly line Daily operations within a business Building a shopping mall Image and Signal Processing Entertainment (Image Rendering) Database and Data Mining

References Tutorials located in the Maui High Performance Computing Center's "SP Parallel Programming Workshop". Tutorials located at the Cornell Theory Center's "Education and Training" web page. "Designing and Building Parallel Programs". Ian FosterCarriero, Nicholas and Gelernter, David, "How to Write Parallel Programs - A First Course". MIT Press, Cambridge, Massachusetts. Dowd, Kevin, High Performance Computing", O'Reilly & Associated, Inc., Sebastopol, California. Hockney, R.W. and Jesshope, C.R., "Parallel Computers 2",Hilger, Bristol and Philadelphia. Ragsdale, Susan, ed., "Parallel Programming", McGraw-Hill, Inc., New York. Chandy, K. Mani and Taylor, Stephen, "An Introduction to Parallel Programming", Jones and Bartlett, Boston. www.wikipedia.org

THANK YOU