Parallel Computers Chapter 1.

Parallel Computers Chapter 1

The Demand for Computational Speed
Introduction There is a continual demand for greater computational speed from a computer system than is currently possible Areas requiring great computational speed include numerical modeling and simulation of scientific and engineering problems. Computations must be completed within a “reasonable” time period

The Demand for Computational Speed
Grand Challenge Problems One that cannot be solved in a reasonable amount of time with today’s computers. Obviously, an execution time of 10 years is always unreasonable. Examples: Modeling large DNA structures Global weather forecasting Modeling motion of astronomical bodies

Grand Challenge Problems
Weather Forecasting Atmosphere modeled by dividing it into 3-dimensional cells. Calculations of each cell repeated many times to model passage of time

Example: Global Weather Forecasting
Suppose whole global atmosphere divided into cells of size 1 mile  1 mile  1 mile to a height of 10 miles (10 cells high) - about 5  108 cells. Suppose each calculation requires 200 floating point operations. In one time step, 1011 floating point operations necessary. To forecast the weather over 7 days using 1-minute intervals, a computer operating at 1Gflops (109 floating point operations/s) takes 106 seconds or over 10 days. To perform calculation in 5 minutes requires computer operating at 3.4 Tflops (3.4  1012 floating point operations/sec).

Example: Modeling Motion of Astronomical Bodies
Each body attracted to each other body by gravitational forces. Movement of each body predicted by calculating total force on each body. With N bodies, N - 1 forces to calculate for each body, or approx. N2 calculations. (N log2 N for an efficient approx. algorithm.) After determining new positions of bodies, calculations repeated.

A galaxy might have, say, 1011 stars. Even if each calculation done in 1 ms (extremely optimistic figure), it takes 109 years for one iteration using N2 algorithm and almost a year for one iteration using an efficient N log2 N approximate algorithm

Sample Output Could look like this:

Parallel Computing What is it? What is the Motive?
Using more than one computer, or a computer with more than one processor, to solve a problem. What is the Motive? Usually faster computation - very simple idea - that n computers operating simultaneously can achieve the result n times faster - it will not be n times faster for various reasons. Other motives include: fault tolerance, larger amount of memory available, ...

Parallel Computing History – When did this all begin?:
Parallel computers - computers with more than one processor - and their programming - parallel programming - has been around for more than 40 years.

Gill writes in 1958: “... There is therefore nothing new in the idea of parallel programming, but its application to computers. The author cannot believe that there will be any insuperable difficulty in extending it to computers. It is not to be expected that the necessary programming techniques will be worked out overnight. Much experimenting remains to be done. After all, the techniques that are commonly used in programming today were only won at the cost of considerable toil several years ago. In fact the advent of parallel programming may do something to revive the pioneering spirit in programming which seems at the present to be degenerating into a rather dull and routine occupation ...” Gill, S. (1958), “Parallel Programming,” The Computer Journal, vol. 1, April, pp

2. Potential for Increased Computational Speed
Notation: Now and in the future, the number of processors will be identified as p. We will also use the term “multicomputer” to include all parallel computer systems with more than one processor. Now, on to some definitions:

2.1 Speedup Factor where ts is execution time on a single processor and tp is execution time on a multiprocessor. S(p) gives increase in speed by using multiprocessor. Use best sequential algorithm with single processor system. Underlying algorithm for parallel implementation might be (and is usually) different.

2.1 Speedup Factor Speedup factor can also be cast in terms of computational steps: Can also extend time complexity to parallel computations.

2.1 Speedup Factor Maximum speedup is usually p with p processors (linear speedup). It is possible to get superlinear speedup (greater than p) but usually a specific reason such as: Extra memory in multiprocessor system Nondeterministic algorithm There is also the definition of Efficiency

2.2 What is the Maximum Speedup?
Several factors will appear as overhead in the parallel version and limit the speedup, notably: Periods when not all of the processors can be performing useful work and are simply idle. Extra computations (or duplicate ones) in the parallel version Communication time

Speedup factor is given by: This equation is known as Amdahl’s law

Even with infinite number of processors, maximum speedup limited to 1/f . Example: With only 5% of computation being serial, maximum speedup is 20, irrespective of number of processors.

Superlinear Speedup example - Searching
Searching each sub-space sequentially

Searching each sub-space in parallel

Speed-up then given by

The worst case for sequential search is when the solution is found in the last sub-space search. Then parallel version offers greatest benefit, i.e.

Least advantage for parallel version is when solution found in first sub-space search of the sequential search, i.e. Actual speed-up depends upon which subspace holds solution but could be extremely large

One last definition is Scalability. This is an overloaded word. Two types we are interested in are: Architecture or Hardware scalability Algorithmic scalability

2.3 Message-Passing Computations
Message passing can be a very significant overhead portion of parallel computation tp = tcomm+tcomp The following ratio can be used as a comparison metric for algorithms (tcomp)/ (tcomm)

3. Types of Parallel Computers
Now that we agree there is some potential for speedup, let us explore how a parallel machine might be constructed Two principal types: Shared memory multiprocessor Distributed memory multicomputer

Conventional Computer
Consists of a processor executing a program stored in a (main) memory: Each main memory location located by its address. Addresses start at 0 and extend to 2b - 1 when there are b bits (binary digits) in address.

3.1 Shared Memory Multiprocessor System
The Natural way to extend single processor model - have multiple processors connected to multiple memory modules, such that each processor can access any memory module :

3.1 Shared Memory Multiprocessor System
A Simplistic view of a small shared memory multiprocessor Examples: Dual Pentiums Quad Pentiums

Quad Pentium Shared Memory Multiprocessor

Programming Shared Memory Multiprocessors
Threads - programmer decomposes program into individual parallel sequences, (threads), each being able to access variables declared outside threads. Example Pthreads Sequential programming language with preprocessor compiler directives to declare shared variables and specify parallelism. Example OpenMP - industry standard - needs OpenMP compiler

Programming Shared Memory Multiprocessors
Sequential programming language with added syntax to declare shared variables and specify parallelism. Example UPC (Unified Parallel C) - needs a UPC compiler. Parallel programming language with syntax to express parallelism - compiler creates executable code for each processor (not now common) Sequential programming language and ask parallelizing compiler to convert it into parallel executable code. - also not now common

3.2 Message-Passing Multicomputer
Complete computers connected through an interconnection network:

There are a lot of definitions that are used to compare networks Bandwidth Latency (network, and message) Cost Bisection width

Interconnection Networks Limited and exhaustive interconnections 2- and 3-dimensional meshes Hypercube (not now common) Using Switches: Crossbar Trees Multistage interconnection networks

Two-dimensional array (mesh)
There are also three-dimensional - used in some large high performance systems

Three-dimensional hypercube

Four-dimensional hypercube
Hypercubes were popular in the 1980’s

Crossbar switch

Multistage Interconnection Network Example: Omega network

Communication Methods
When routing messages you have two ways to transfer messages Circuit switching Packet switching There are issues to watch out for in routing algorithms Deadlock Livelock

3.3 Distributed Shared Memory
Making main memory of group of interconnected computers look as though a single memory with single address space. Then you can use shared memory programming techniques

3.4 MIMD and SIMD Classifications
Flynn (1966) created a classification for computers based upon instruction streams and data streams: Single instruction stream-single data stream (SISD) computer Single processor computer - single stream of instructions generated from program. Instructions operate upon a single stream of data items

Multiple Instruction Stream-Multiple Data Stream (MIMD) Computer General-purpose multiprocessor system - each processor has a separate program and one instruction stream is generated from each program for each processor. Each instruction operates upon different data. Both the shared memory and the message-passing multiprocessors so far described are in the MIMD classification.

Single Instruction Stream-Multiple Data Stream (SIMD) Computer A specially designed computer - a single instruction stream from a single program, but multiple data streams exist. Instructions from program broadcast to more than one processor. Each processor executes same instruction in synchronism, but using different data. Developed because a number of important applications that mostly operate upon arrays of data.

Multiple Program Multiple Data (MPMD) Structure Within the MIMD classification, each processor will have its own program to execute:

Single Program Multiple Data (SPMD) Structure Single source program written and each processor executes its personal copy of this program, although independently and not in synchronism. Source program can be constructed so that parts of the program are executed by certain computers and not others depending upon the identity of the computer

4. Cluster Computing Networked Computers as a Computing Platform.
These became an attractive alternative to expensive supercomputers in the early 1990’s There were several early projects. Berkeley NOW Project (network of workstations) NASA Beowulf project.

4. Cluster Computing Key Advantages
Very High performance workstations and PC’s readily available at low cost. The latest processors can easily be incorporated into the system as they become available. Existing software can be used or modified.

4.2 Cluster Configurations
Existing Networked Computers Like the ECC lab. You need to be kind to the interactive users (those at the console) Problem – they can shut off one machine and mess up your computation.

Beowulf Clusters A group of interconnected “commodity” computers achieving high performance with low cost. Typically using commodity interconnects (high speed Ethernet), and Linux OS The Beowulf project started at NASA Goddard in 1993

Beyond Beowulf Cluster Interconnects Originally Fast Ethernet Now Gigabit Ethernet More Specialized Interconnects Myrinet – 2.4Gbits/sed cLan SCI QNet Infiniband In my opinion, this is the one to watch!

4.3 Setting up a Cluster Hardware Configuration

4.3 Setting up a Cluster Software Tools (Middleware) PVM – MPI
Developed in the late 1980’s MPI Standard defined in the late 1990’s

5. Summary This chapter introduced the following concepts:
Parallel computers and programming Speedup and other factors Extension of a single processor system into a shared memory multiprocessor The message passing multiprocessor Interconnection networks suitable for message passing Networked workstations as a parallel programming platform Cluster computing.

Message Passing Computing
Chapter 2

Introduction In this chapter, we outline the basic concepts in message-passing computing. We introduce the basic structure of message-passing programs and how to specify message passing between processes. We discuss these in general, and then we outline two specific systems, PVM and MPI Finally, we discuss how to evaluate message-passing parallel programs, both theoretically and in practice.

Basics of Message Passing Programming
1.1 Programming Options Programming a message-passing multicomputer can be achieved by 1. Designing a special parallel programming language 2. Extending the syntax/reserved words of an existing sequential high-level language to handle message passing 3. Using an existing sequential high-level language and providing a library of external procedures for message passing

Basics of Message Passing Programming
We will concentrate upon the third option. Necessary to say explicitly what processes are to be executed, when to pass messages between concurrent processes, and what to pass in the messages. Two primary methods are needed in this form of a message-passing system: 1. A method of creating separate processes for execution on different computers 2. A method of sending and receiving messages

1.2 Process Creation MPMD model

1.2 Process Creation SPMD model
Different processes are merged into one program. Within the program are control statements that will customize the code; i.e. select different parts for each process

1.2 Process Creation MPMD model Separate programs for each processor. One processor executes master process. Other processes started from within master process - dynamic process creation.

1.3 Message-Passing Routines
Passing a message between processes using send() and recv() library calls:

Synchronous Message Passing Routines that actually return when the message transfer has been completed. Synchronous send routine Waits until complete message can be accepted by the receiving process before sending the message. Synchronous receive routine Waits until the message it is expecting arrives. Synchronous routines intrinsically perform two actions: They transfer data and they synchronize processes.

Synchronous send() and recv() using 3-way protocol

Asynchronous Message Passing Routines that do not wait for actions to complete before returning. Usually require local storage for messages. More than one version depending upon the actual semantics for returning. In general, they do not synchronize processes but allow processes to move forward sooner. Must be used with care.

Blocking and Nonblocking Message Passing Blocking - has been used to describe routines that do not return until the transfer is completed. The routines are “blocked” from continuing. In that sense, the terms synchronous and blocking were synonymous. Non-blocking - has been used to describe routines that return whether or not the message had been received. The terms blocking and nonblocking redefined in systems such as MPI:

MPI Definitions of Blocking and Non-Blocking Blocking - return after their local actions complete, though the message transfer may not have been completed. Non-blocking - return immediately. Assumed that the data storage being used for the transfer is not modified by the subsequent statements prior to the data storage being used for the transfer, and it is left to the programmer to ensure this.

How message-passing routines can return before the message transfer has been completed Generally, a message buffer needed between source and destination to hold message:

Once local actions completed and message is safely on its way, sending process can continue with subsequent work. Buffers only of finite length and a point could be reached when send routine held up because all available buffer space exhausted. Then, send routine will wait until storage becomes re-available - i.e then routine behaves as a synchronous routine.

Message Selection Message Tag Used to differentiate between different types of messages being sent. Message tag is carried within message. If special type matching is not required, a wild card message tag is used, so that the recv() will match with any send().

Example: To send a message, x, with message tag 5 from a source process, 1, to a destination process, 2, and assign to y:

“Group” message passing routines Have routines that send message(s) to a group of processes or receive message(s) from a group of processes Higher efficiency than separate point-to-point routines although not absolutely necessary.

Broadcast Sending the same message to all the processes concerned with the problem. Multicast - sending the same message to a defined group of processes.

Scatter Sending each element of an array of data in the root to a separate process. The contents of the ith location of the array is sent to the ith process.

Gather Having one process collect individual values from a set of processes.

Reduce Gather operation combined with a specified arithmetic or logical operation. Example, the values could be gathered and then added together by the root:

2. Using Workstation Clusters
2.1 Software Tools PVM (Parallel Virtual Machine) - Perhaps the first widely adopted attempt at using a workstation cluster as a multicomputer platform developed by Oak Ridge National Laboratories. Provides for a software environment for message passing between homogeneous or heterogeneous computers and has a collection of library routines that the user can employ with C or FORTRAN programs. Available at no charge.

Message routing between computers done by PVM daemon processes installed by PVM on computers that form the virtual machine.

2.2 MPI Message passing library standard developed by group of academics and industrial partners to foster more widespread use and portability. Defines routines, not implementation. Several free implementations exist.

2.2 MPI Process Creation and Execution Purposely not defined and will depend upon the implementation. Only static process creation is supported in MPI version 1. All the processes must be defined prior to execution and started together. Use SPMD model of computation.

Unsafe message passing Example

2.2 MPI Communicators Defines the scope of a communication operation. Processes have ranks associated with the communicator. Initially, all processes enrolled in a “universe” called MPI_COMM_WORLD, and each process is given a unique rank, a number from 0 to n - 1, where there are n processes. Other communicators can be established for groups of processes. Communication domains of libraries can be separated from that of a user program

Using the SPMD Computation Model
main (int argc, char *argv[ ]) { MPI_Init(&argc, &argv); . MPI_Comm_rank(MPI_COMM_WORLD, &myrank); /* find process rank */ if (myrank == 0) master(); else slave(); MPI_Finalize(); }

2.2 MPI Global and Local Variables Any global declarations of variables will be duplicated in each process. Variables that are not to be duplicated will need to be declared within code only executed by that process.

2.2 MPI Blocking Routines Return when they are locally complete - when the location used to hold the message can be used again or altered without affecting the message being sent. A blocking send will send the message and return. This does not mean that the message has been received, just that the process is free to move on without adversely affecting the message.

2.2 MPI Parameters of blocking send

2.2 MPI Parameters of blocking receive

2.2 MPI To send an integer x from process 0 to process 1, MPI_Comm_rank(MPI_COMM_WORLD,&myrank); /* find rank */ if (myrank == 0) { int x; MPI_Send(&x, 1, MPI_INT, 1, msgtag, MPI_COMM_WORLD); } else if (myrank == 1) { MPI_Recv(&x, 1, MPI_INT, 0,msgtag,MPI_COMM_WORLD,status); }

2.2 MPI MPI Nonblocking Routines Nonblocking send - MPI_Isend() - will return “immediately” even before source location is safe to be altered. Nonblocking receive - MPI_Irecv() - will return even if no message to accept.

2.2 MPI Nonblocking Routine Formats MPI_Isend(buf,count,datatype,dest,tag,comm,request) MPI_Irecv(buf,count,datatype,source,tag,comm, request) Completion detected by MPI_Wait() and MPI_Test(). MPI_Wait() waits until operation completed and returns then. MPI_Test() returns with flag set indicating whether operation completed at that time. Need to know whether particular operation completed. Determined by accessing request parameter.

2.2 MPI To send an integer x from process 0 to process 1 and allow process 0 to continue, MPI_Comm_rank(MPI_COMM_WORLD, &myrank);/* find rank */ if (myrank == 0) { int x; MPI_Isend(&x,1,MPI_INT, 1, msgtag, MPI_COMM_WORLD, req1); compute(); MPI_Wait(req1, status); } else if (myrank == 1) { MPI_Recv(&x,1,MPI_INT,0,msgtag, MPI_COMM_WORLD, status); }

2.2 MPI Four Send Communication Modes Standard Mode Send - Not assumed that corresponding receive routine has started. Amount of buffering not defined by MPI. If buffering provided, send could complete before receive reached. Buffered Mode - Send may start and return before a matching receive. Necessary to specify buffer space via routine MPI_Buffer_attach().

2.2 MPI Four Send Communication Modes Synchronous Mode - Send and receive can start before each other but can only complete together. Ready Mode - Send can only start if matching receive already reached, otherwise error. Use with care.

2.2 MPI Four Send Communication Modes Each of the four modes can be applied to both blocking and nonblocking send routines. Only the standard mode is available for the blocking and nonblocking receive routines. Any type of send routine can be used with any type of receive routine.

2.2 MPI Collective Communication Involves set of processes, defined by an intra-communicator. Message tags not present. Principal collective operations: MPI_Bcast() - Broadcast from root to all other processes MPI_Gather() - Gather values for group of processes MPI_Scatter() - Scatters buffer in parts to group of processes

2.2 MPI Collective Communication MPI_Alltoall() - Sends data from all processes to all processes MPI_Reduce() - Combine values on all processes to single value MPI_Reduce_scatter() - Combine values and scatter results MPI_Scan() - Compute prefix reductions of data on processes

2.2 MPI Collective Communication Example:To gather items from group of processes into process 0, using dynamically allocated memory in root process: MPI_Gather() gathers from all processes, including root.

int data[10]; /*data to be gathered from processes*/ MPI_Comm_rank(MPI_COMM_WORLD, &myrank); /* find rank */ if (myrank == 0) { MPI_Comm_size(MPI_COMM_WORLD, &grp_size); /*find group size*/ buf = (int *)malloc(grp_size*10*sizeof (int)); /*allocate memory*/ } MPI_Gather(data,10,MPI_INT,buf,grp_size*10,MPI_INT,0,MPI_COMM_WORLD) ;

2.2 MPI Barrier As in all message-passing systems, MPI provides a means of synchronizing processes by stopping each one until they all have reached a specific “barrier” call.

#include “mpi.h” #include <stdio.h> #include <math.h> #define MAXSIZE 1000 void main(int argc, char *argv) { int myid, numprocs; int data[MAXSIZE], i, x, low, high, myresult, result; char fn[255]; char *fp; MPI_Init(&argc,&argv); MPI_Comm_size(MPI_COMM_WORLD,&numprocs); MPI_Comm_rank(MPI_COMM_WORLD,&myid); if (myid == 0) { /* Open input file and initialize data */ strcpy(fn,getenv(“HOME”)); strcat(fn,”/MPI/rand_data.txt”); if ((fp = fopen(fn,”r”)) == NULL) { printf(“Can’t open the input file: %s\n\n”, fn); exit(1); } for(i = 0; i < MAXSIZE; i++) fscanf(fp,”%d”, &data[i]); MPI_Bcast(data, MAXSIZE, MPI_INT, 0, MPI_COMM_WORLD); /* broadcast data */ x = n/nproc; /* Add my portion Of data */ low = myid * x; high = low + x; for(i = low; i < high; i++) myresult += data[i]; printf(“I got %d from %d\n”, myresult, myid); /* Compute global sum */ MPI_Reduce(&myresult, &result, 1, MPI_INT, MPI_SUM, 0, MPI_COMM_WORLD); if (myid == 0) printf(“The sum is %d.\n”, result); MPI_Finalize();

3. Evaluating Parallel Programs
3.1 Equations for Parallel Execution Time Sequential execution time, ts: Estimate by counting computational steps of best sequential algorithm. Parallel execution time, tp: In addition to number of computational steps, tcomp, need to estimate communication overhead, tcomm: tp = tcomp + tcomm

Computational Time Count number of computational steps. When more than one process executed simultaneously, count computational steps of most complex process. Generally, function of n and p, i.e. tcomp = f (n, p) Often break down computation time into parts. Then tcomp = tcomp1 + tcomp2 + tcomp3 + … Analysis usually done assuming that all processors are same and operating at same speed.

Communication Time Many factors, including network structure and network contention. As a first approximation, use tcomm = tstartup + ntdata tstartup is startup time, essentially time to send a message with no data. Assumed to be constant. tdata is transmission time to send one data word, also assumed constant, and there are n data words.

Idealized Communication Time

Benchmark Factors With ts, tcomp, and tcomm, can establish speedup factor and computation/communication ratio for a particular algorithm/implementation:

Both functions of number of processors, p, and number of data elements, n. Will give indication of scalability of parallel solution with increasing number of processors and problem size. Computation/communication ratio will highlight effect of communication with increasing problem size and system size.

Important Note on Interpretation of Equations Only intended to give a starting point to how an algorithm might perform in practice. Parallel execution time, tp, normalized in units of an arithmetic operation, which will depend upon computer system. We might find that the computation requires m computational steps so that tcomp = m

Since we are measuring time in units of computational steps, the communication time has to be measured in the same way. All data types are assumed to require the same time. Suppose q messages are sent, each containing n data items. We have tcomm = q(tstartup + ntdata)

Latency Hiding A way to ameliorate situation of significant message communication times is to overlap communication with subsequent computations. Nonblocking send routines provided particularly to enable latency hiding. Latency hiding can also be achieved by mapping multiple processes on a processor and use a time-sharing facility that switches for one process to another when the first process is stalled because of incomplete message passing or otherwise. Relies upon an efficient method of switching from one process to another. Threads offer an efficient mechanism.

3.2 Time Complexity As with sequential computations, a parallel algorithm can be evaluated through the use of time complexity (notably the O notation — “order of magnitude,” big-oh). Start with an estimate of the number of the computational steps, considering all arithmetic and logical operations to be equal and ignoring other aspects of the computation such as computational tests. An expression of the number of computational steps is derived, often in terms of the number of data items being handled by the algorithm.

Formal Definition The O notation f(x) = O(g(x)) if and only if there exists positive constants, c and x0, such that 0 £ f(x) £ cg(x) for all x ³ x0 where f(x) and g(x) are functions of x. For example, if f(x) = 4x2 + 2x + 12, the constant c = 6 would work with the formal definition to establish that f(x) = O(x2), since 0 < 4x2 + 2x + 12 £ 6x2 for x ³ 3. Alternative functions for g(x) that will satisfy definition. Use the function that grows least for g(x).

Q notation - upper bound f(x) = Q(g(x)) if and only if there exists positive constants c1, c2, and x0 such that 0 £ c1g(x) £ f(x) £ c2g(x) for all x ³ x0. If f(x) = Q(g(x)), it is clear that f(x) = O(g(x)) is also true. W notation - lower bound f(x) = W(g(x)) if and only if there exists positive constants c and x0 such that 0 £ cg(x) £ f(x) for all x ³ x0. It follows from this definition that f(x) = 4x2 + 2x + 12 = W(x2) We can read O() as “grows at most as fast as” and W() as “grows at least as fast as.”

Example The execution time of a sorting algorithm often depends upon the original order of the numbers to be sorted. It may be that it requires at least n log n steps, but could require n2 steps for n numbers depending upon the order of the numbers. This would be indicated by a time complexity of W(n log n) and O(n2).

Time Complexity of a Parallel Algorithm Time complexity analysis hides lower terms - tcomm has time complexity of O(n). Time complexity of tp will be sum of complexity of computation and communication. Example To add n numbers on two computers, where each computer adds n/2 numbers together, and the numbers are initially all held by first computer. Second computer submits its result to first computer for adding the two partial sums together.

Several phases: 1. Computer 1 sends n/2 numbers to computer 2. 2. Both computers add n/2 numbers simultaneously. 3. Computer 2 sends its partial result back to computer 1. 4. Computer 1 adds the partial sums to produce the final result. Computation (for steps 2 and 4): tcomp = n/2 + 1 Communication (for steps 1 and 3): tcomm = (tstartup + n/2tdata) + (tstartup + tdata) = 2tstartup + (n/2 + 1)tdata The computational complexity is O(n). The communication complexity is O(n). The overall time complexity is O(n).

Cost-Optimal Algorithms A cost-optimal (or work-efficient or processor-time optimality) algorithm is one in which the cost to solve a problem is proportional to the execution time on a single processor system (using the fastest known sequential algorithm); i.e., Cost = tp x n = k x ts where k is a constant. Given time complexity analysis, we can say that a parallel algorithm is cost-optimal algorithm if (Parallel time complexity) ´ (number of processors) = sequential time complexity

Example Suppose the best known sequential algorithm for a problem has time complexity of O(n log n). A parallel algorithm for the same problem that uses n processes and has a time complexity of O(log n) is cost optimal, whereas a parallel algorithm that uses n2 processors and has time complexity of O(1) is not cost optimal.

4. Debugging and Evaluating Parallel Programs Empirically
4.2 Visualization Tools Programs can be watched as they are executed in a space-time diagram (or process-time diagram):

4.3 Debugging Strategies Geist et al. (1994a) suggest a three-step approach to debugging message-passing programs: 1. If possible, run the program as a single process and debug as a normal sequential program. 2. Execute the program using two to four multitasked processes on a single computer. Now examine actions such as checking that messages are indeed being sent to the correct places. It is very common to make mistakes with message tags and have messages sent to the wrong places. 3. Execute the program using the same two to four processes but now across several computers. This step helps find problems that are caused by network delays related to synchronization and timing.

4.4 Evaluating Programs Empirically Measuring Execution Time To measure the execution time between point L1 and point L2 in the code, we might have: . L1: time(&t1); /* start timer */ L2: time(&t2); /* stop timer */ elapsed_time = difftime(t2, t1); /*elapsed_time=t2-t1*/ printf(“Elapsed time = %5.2f seconds”, elapsed_time); MPI provides the routine MPI_Wtime() for returning time (in seconds).

Communication Time by the Ping-Pong Method
. L1: time(&t1); send(&x, P1); recv(&x, P1); L2: time(&t2); elapsed_time = 0.5 * difftime(t2, t1); printf(“Elapsed time = %5.2f seconds”, elapsed_time); P1 . recv(&x, P0); send(&x, P0);

Profiling A profile is a histogram or graph showing time spent on different parts of program:

5. Summary This chapter introduces the following concepts:
Basic Message Passing techniques Send, Receive, and collective operations Software tools for harnessing a network of workstations Modeling communication Communication latency and latency hiding Time complexity of parallel algorithms Debugging and evaluating parallel programs

Embarrassingly Parallel Computations
Chapter 3

1. Ideal Parallel Computation
In this chapter, we will consider applications where there is minimal interaction between the slave processes. Slaves may be statically assigned or dynamically assigned. Load balancing techniques can offer improved performance. We will introduce simple load balancing here Chapter 7 will cover load balancing when the slaves must interact.

Parallelizing these should be obvious and requires no special techniques or algorithms. A truly embarrassingly parallel computation requires no communication. (SPMD Model)

Nearly embarrassingly parallel computations Require results to be distributed and collected in some way. A common approach is the Master-Slave organization.

2. Embarrassingly Parallel Examples
2.1 Geometrical Transformation of Images The most basic way to store a computer image is a pixmap (each pixel is stored in a 2D array) Geometrical transformations require mathematical operations to be performed on the coordinates of each pixel without affecting its value.

2.1 Geometrical Transformation of Images
There are several different transformations Shifting Object is shifted Dx in the x dimension and Dy in the y dimension: x’=x+ Dx y’=y+ Dy Where x and y are the original and x’ and y’ are the new coordinates. Scaling Object scaled by factor Sx in the x-direction and Sy in the y-direction x’=xSx y’=ySy

Rotation Object rotated through an angle q about the origin of the coordinate system: x’=x cosq + y sinq y’=-x sinq + y cosq Clipping Deletes from the displayed picture those points outside the defined area.

The input data is the pixmap (bitmap) Typically held in a file, then copied into an array. The master is concerned with dividing up the array among the processors. There are two methods: Blocks Stripes

2.1 Geometrical Transformation of Images: Master – pseudo code
for (i = 0, row = 0; i < 48; i++, row = row + 10) /* for each process*/ send(row, Pi); /* send row no.*/ for (i = 0; i < 480; i++) /* initialize temp */ for (j = 0; j < 640; j++) temp_map[i][j] = 0; for (i = 0; i < (640 * 480); i++) { /* for each pixel */ recv(oldrow,oldcol,newrow,newcol, PANY); /* accept new coords */ if !((newrow < 0)||(newrow >= 480)||(newcol < 0)||(newcol >= 640)) temp_map[newrow][newcol]=map[oldrow][oldcol]; } for (i = 0; i < 480; i++) /* update bitmap */ map[i][j] = temp_map[i][j];

2.1 Geometrical Transformation of Images: Slave – pseudo code
recv(row, Pmaster); /* receive row no. */ for (oldrow = row; oldrow < (row + 10); oldrow++) for (oldcol = 0; oldcol < 640; oldcol++) {/* transform coords */ newrow = oldrow + delta_x; /* shift in x direction */ newcol = oldcol + delta_y; /* shift in y direction */ send(oldrow,oldcol,newrow,newcol, Pmaster);/* coords to master */ }

2.1 Geometrical Transformation of Images: Analysis
Sequential: If each pixel required 1 time step then ts = n2 So, time complexity is O(n2) Recall that for parallel tp = tcomp + tcomm tcomm = tstartup + mtdata = O(m) tcomp = 2(n2/p) = O(n2/p) tp = O(n2) What’s the problem?

2.1 Geometrical Transformation of Images: Analysis
The constants for the communication (for parallel) far exceed the constants in the computation (for sequential) Makes it relatively impractical This problem is probably better suited for a shared memory system.

2.2 Mandelbrot Set Def: set of points in the plane that are quasi-stable when computed by iterating the function: zk+1 = zk2+c Note: z is complex, and c is complex. The initial value for z is 0 Iterations continue until the magnitude of z is greater than 2 or the number of iterations reaches some limit

2.2 Mandelbrot Set

2.2 Mandelbrot Set: Sequential
int cal_pixel(complex c) { int count, max; complex z; float temp, lengthsq; max = 256; z.real = 0; z.imag = 0; count = 0; /* number of iterations */ do { temp = z.real * z.real - z.imag * z.imag + c.real; z.imag = 2 * z.real * z.imag + c.imag; z.real = temp; lengthsq = z.real * z.real + z.imag * z.imag; count++; } while ((lengthsq < 4.0) && (count < max)); return count; }

2.2 Mandelbrot Set: Parallel
Static Task Assignment – Master for (i = 0, row = 0; i < 48; i++, row = row + 10) /* for each process*/ send(&row, Pi); /* send row no.*/ for (i = 0; i < (480 * 640); i++) { /* from processes, any order */ recv(&c, &color, PANY); /*receive coords & colors */ display(c, color); /* display pixel on screen */ }

2.2 Mandelbrot Set: Slave recv(&row, Pmaster); /* receive row no. */
for (x = 0; x < disp_width; x++) /* screen coordinates x and y */ for (y = row; y < (row + 10); y++) { c.real = min_real + ((float) x * scale_real); c.imag = min_imag + ((float) y * scale_imag); color = cal_pixel(c); send(&c, &color, Pmaster);/* send coords, color to master */ }

2.2 Mandelbrot Set Dynamic Task Assignment Work Pool/Processor Farms

Master count = 0; /* counter for termination*/
row = 0; /* row being sent */ for (k = 0; k < procno; k++) { /* assuming procno<disp_height */ send(&row, Pk, data_tag); /* send initial row to process */ count++; /* count rows sent */ row++; /* next row */ } do { recv (&slave, &r, color, PANY, result_tag); count--; /* reduce count as rows received */ if (row < disp_height) { send (&row, Pslave, data_tag); /* send next row */ count++; } else send (&row, Pslave, terminator_tag); /* terminate */ rows_recv++; display (r, color); /* display row */ } while (count > 0);

2.2 Mandelbrot Set: Slave recv(y, Pmaster, ANYTAG, source_tag);
/* receive 1st row to compute */ while (source_tag == data_tag) { c.imag = imag_min + ((float) y * scale_imag); for (x = 0; x < disp_width; x++) { /* compute row colors */ c.real = real_min + ((float) x * scale_real); color[x] = cal_pixel(c); } send(&i, &y, color, Pmaster, result_tag); /* row colors to master */ recv(y, Pmaster, source_tag); /* receive next row */ };

2.2 Mandelbrot Set

2.2 Mandelbrot Set: Analysis
Sequential ts= max x n = O(n) Parallel Phase 1: Communication Out tcomm=s(tstartup+tdata) It could be possible to use a scatter routine here which would cut the number of startup times. Phase 2: Computation (max x n)/s Phase 3: Communication Back in tcomm2= (n/s)(tstartup+tdata) Overall tp <= (max x n)/s + (n/s + s)(tstartup+tdata)

2.3 Monte Carlo Methods The basis of Monte Carlo methods is the use of random selections in calculations. Example – Calculate p A circle is formed within a square The circle has unit radius Therefore the square has sides of length 2 Area of the square is 4

2.3 Monte Carlo Methods

2.3 Monte Carlo Methods The ratio of the area of the circle to the square is (p r2)/2x2 = p/4 Points within the square are chosen randomly and a score is kept of how many points happen to lie within the circle The fraction of points within the circle will be p/4, given a sufficient number of randomly selected samples.

2.3 Monte Carlo Methods Random Number Generation
The most popular way of creating a pueudorandom number sequence: x1, x2, x3, …, xi-1, xi, xi+1, …, xn-1, xn, is by evaluating xi+1 from a carefully chosen function of xi, often of the form xi+1 = (axi + c) mod m where a, c, and m are constants chosen to create a sequence that has similar properties to truly random sequences.

2.3 Monte Carlo Methods Parallel Random Number Generation
It turns out that xi+1 = (axi + c) mod m xi+k = (Axi + C) mod m where A = ak mod m, C = c(ak-1 + an-2 + … + a1 + a0) mod m, and k is a selected “jump” constant.

5. Summary This Chapter introduced the following concepts:
An ideal embarrassingly parallel computation Embarrassingly parallel problems and analyses Partitioning a two-dimensional data set Work pool approach to achieve load balancing Counter termination algorithm Monte Carlo methods Parallel random number generation

Partitioning and Divide-and-Conquer Strategies
Chapter 4

Introduction In this chapter, we explore two of the most fundamental techniques in parallel programming: partitioning and divide-and-conquer. In partitioning, the problem is simply divided into separate parts and each part is computed separately. Divide and conquer usually applies partitioning in a recursive manner by continually dividing the problem into smaller and smaller parts and combining the results.

1. Partitioning 1.1 Partitioning Strategies
This is the basis for all parallel programming, in one form or another. Embarrassingly parallel problems of the last chapter used partitioning. Most partitioning formulations, however, require the results to be combined

1. Partitioning Partitioning can be applied to the program data
this is called data partitioning or domain decomposition Partitioning can also be applied to the functions of the program this is called functional decomposition It is much less common to find concurrent functions in a program, but data partitioning is a main strategy for parallel programming.

1. Partitioning Consider the problem of adding n numbers on m processors. Each processor would add n/m numbers into a partial sum

1. Partitioning How would you do this in a master-slave approach?
How would the slaves get the numbers? One send per slave? One broadcast?

1. Partitioning Send-Receive Code: Master:
s = n/m; /* number of numbers for slaves*/ for (i = 0, x = 0; i < m; i++, x = x + s) send(&numbers[x], s, P i ); /* send s numbers to slave */ sum = 0; for (i = 0; i < m; i++) { /* wait for results from slaves */ recv(&part_sum, P_ANY ); sum = sum + part_sum; /* accumulate partial sums */ }

1. Partitioning Slave recv(numbers, s, P_master );
/* receive s numbers from master */ part_sum = 0; for (i = 0; i < s; i++) /* add numbers */ part_sum = part_sum + numbers[i]; send(&part_sum, P_master ); /* send sum to master */

1. Partitioning Broadcast Code Master:
s = n/m; /* number of numbers for slaves */ bcast(numbers, s, P slave_group );/* send all numbers to slaves */ sum = 0; for (i = 0; i < m; i++){ /* wait for results from slaves */ recv(&part_sum, P ANY ); sum = sum + part_sum; /* accumulate partial sums */ }

1. Partitioning Broadcast Code Slave
bcast(numbers, s, P master ); /* receive all numbers from master*/ start = slave_number * s; /* slave number obtained earlier */ end = start + s; part_sum = 0; for (i = start; i < end; i++) /* add numbers */ part_sum = part_sum + numbers[i]; send(&part_sum, P master ); /* send sum to master */

1. Partitioning Analysis Phase 1 - Communication (numbers to slaves)
tcomm1 = m(tstartup + (n/m)tdata) tcomm1 = tstartup + ntdata = O(n) Phase 2 - Computation tcomp1 = n/m -1 Phase 3 - Communication using individual sends: tcomm2 = m(tstartup + tdata) using a gather - reduce: tcomm2 = tstartup + mtdata Phase 4 - Computation tcomp2 = m -1

1. Partitioning Analysis (cont.) Overall
tp = (m+1)tstartup +(n+m)tdata + m + n/m tp = O(n+m) We see that the parallel time complexity is worse than the sequential time complexity O(n)

1.2 Divide and Conquer Characterized by dividing a problem into subproblems that are of the same form as the larger problem. Further divisions into still smaller sub-problems are usually done by recursion I know that one would never use recursion to add a list of numbers, but....the following discussion is applicable to any problem that is formulated by a recursive divide-and-conquer method.

1.2 Divide and Conquer A sequential recursive definition for adding a list of numbers is int add(int *s) /* add list of numbers, s */ { if (number(s) =< 2) return (n1 + n2);/* see explanation */ else { Divide (s, s1, s2); /* divide s into two parts, s1 and s2 */ part_sum1 = add(s1); /*recursive calls to add sub lists */ part_sum2 = add(s2); return (part_sum1 + part_sum2); }

1.2 Divide and Conquer When each division creates two parts, we form a binary tree.

1.2 Divide and Conquer Parallel Implementation
In a sequential implementation, only one node of the tree can be visited at a time. A parallel solution offers the prospect of traversing several parts of the tree simultaneously. This can be done inefficiently have a processor for each node therefore each processor is only active for one level in the tree computation. Or efficiently re-use processors

1.2 Divide and Conquer

1.2 Divide and Conquer Suppose we statically create eight processors (or processes) to add a list of numbers. Process P 0 /* division phase */ divide(s1, s1, s2); /* divide s1 into two, s1 and s2 */ send(s2, P 4 ); /* send one part to another process */ divide(s1, s1, s2); send(s2, P 2 ); send(s2, P 1 }; part_sum = *s1; /* combining phase */ recv(&part_sum1, P 1 ); part_sum = part_sum + part_sum1; recv(&part_sum1, P 2 ); recv(&part_sum1, P 4 );

1.2 Divide and Conquer Process P 4
recv(s1, P 0 ); /* division phase */ divide(s1, s1, s2); send(s2, P 6 ); send(s2, P 5 ); part_sum = *s1; /* combining phase */ recv(&part_sum1, P 5 ); part_sum = part_sum + part_sum1; recv(&part_sum1, P 6 ); send(&part_sum, P 0 ); Other processes would require similar sequences

1.2 Divide and Conquer Analysis Communication Computation Total
tcomm1 = n/2 tdata +n/4 tdata + n/8 tdata n/p tdata tcomm1 = (n(p-1))/p tdata Computation tcomp = n/p + log p Total If p is constant we end up with O(n) for large n and variable p, we end up with tp = ((n(p-1))/p)tdata +tdata log p + n/p + log p

1.3 M-ary Divide and Conquer
Divide and conquer can also be applied where a task is divided into more than two parts at each stage. Task broken into four parts. The sequential recursive definition would be

int add(int *s) /* add list of numbers, s */ { if (number(s) =< 4) return(n1 + n2 + n3 + n4); else { Divide (s,s1,s2,s3,s4); /* divide s into s1,s2,s3,s4*/ part_sum1 = add(s1); /*recursive calls to add sublists */ part_sum2 = add(s2); part_sum3 = add(s3); part_sum4 = add(s4); return (part_sum1 + part_sum2 + part_sum3 + part_sum4); }

A tree in which each node has four children is called a quadtree.

For example: a digitized image could be divided into four quadrants:

An octree is a tree in which each node has eight children and has application for dividing a three-dimensional space recursively. See Ben Lucchesi’s MS Thesis from 2002 for a parallel collision detection algorithm using octrees.

2. Divide-and-Conquer Examples
2.1 Sorting Using Bucket Sort Bucket sort is not based upon compare-exchange, but is naturally a partitioning method. Bucket sort only works well if the numbers are uniformly distributed across a known interval (say 0 to a-1) This interval will be divided into m regions, and a bucket is assigned to hold all the numbers within each region. The numbers in each bucket will be sorted using a sequential sorting algorithm

2.1 Sorting Using Bucket Sort
Sequential Algorithm place the numbers in their bucket one time step per number -- O(n) Sort the n/m numbers in each bucket O(n/m log (n/m)) If n = km, where k is a constant, we get O(n)

Note: This is a much better than the lower bound for sequential compare and exchange sorting algorithms However, it only applies when the numbers are well distributed.

Parallel Algorithm Clearly, bucket sort can be parallelized by assigning one processor for each bucket.

But -- each processor had to examine each number, so a bunch of wasted effort took place. We can improve this by having m small buckets at each processor and only looking at n/m numbers.

Analysis Phase 1 - Computation and Communication partition numbers into p regions, tcomp1 = n send the data to the slaves, tcomm1 = tstartup + (tdata n) If you can broadcast. Phase 2 - Computation separate n/p number into p small buckets, tcomp2 = n/p Phase 3 - Communication distribute small buckets, tcomm3 = (p-1)(tstartup+(n/p2)tdata) Phase 4 - Computation tcomp4 = (n/p)log(n/p) Overall tp = tstartup +tdatan + n/p + (p-1)(tstartup+(n/p2)tdata)+(n/p)log(n/p)

Phase 3 is basically an all-to-all personalized communication

You can do this in Mpi with MPI_Alltoall()

2.2 Numerical Integration
A general divide-and-conquer technique divides the region continually into parts and lets some optimization function decide when certain regions are sufficiently divided. Let’s look now at our next example: I = òab f (x) dx To integrate (compute the area under the curve) we can divide the area into separate parts, each of which can be calculated by a separate process.

Each region could be calculated using an approximation given by rectangles:

A better approximation can be found by Aligning the rectangles:

Another improvement is to use trapezoids

Static Assignment -- SPMD pseudocode: Process P i if (i == master) { /* read number of intervals required */ scanf(%d”,&n); printf(“Enter number of intervals ”); } bcast(&n, P group ); /* broadcast interval to all processes */ region = (b - a)/p; /* length of region for each process */ start = a + region * i; /* starting x coordinate for process */ end = start + region; /* ending x coordinate for process */ d = (b - a)/n; /* size of interval */ area = 0.0; for (x = start; x < end; x = x + d) area = area + f(x) + f(x+d); area = 0.5 * area * d; reduce_add(&integral, &area, P group ); /* form sum of areas */

Adaptive Quadrature this method allows us to know how far to subdivide. Stop when largest area is approximately = to sum of other 2.

Adaptive Quadrature (cont) Some care might be needed in choosing when to terminate.

2.3 N-Body Problem The objective is to find positions and movements of bodies in space (say planets) that are subject to gravitational forces from other bodies using Newtonian laws of physics. The gravitational force between two bodies of masses ma and mb is given by F = (Gmamb)/(r2) where G is the gravitational constant and r is the distance between the two bodies

2.3 N-Body Problem Subject to forces, a body will accelerate according to Newton’s second law: F = ma where m is the mass of the body, F is the force it experiences, and a is the resultant acceleration.

2.3 N-Body Problem Let the time interval be Dt.
Then, for a particular body of mass m, the force is given by F= (m(vt+1-vt))/Dt and a new velocity vt+1 = vt + (FDt)/m where vt+1 is the velocity of body at time t + 1 and vt is the velocity of body at time t.

2.3 N-Body Problem If a body is moving at a velocity v over the time interval Dt, its position changes by xt+1 -xt = vDt where xt is its position at time t. Once bodies move to new positions, the forces change and the computation has to be repeated.

2.3 N-Body Problem Three-Dimensional Space
In a three-dimensional space having a coordinate system (x, y, z), the distance between the bodies at (xa, ya, za) and (xb, yb, zb) is given by The forces are resolved in the three directions, using, for example,

2.3 N-Body Problem Sequential Code
for (t = 0; t < tmax; t++) /* for each time period */ for (i = 0; i < N; i++) { /* for each body */ F = Force_routine(i); /* compute force on ith body */ v[i]new = v[i] + F * dt / m;/* compute new velocity and x[i]new = x[i] + v[i]new * dt;/* new position (leap-frog) */ } for (i = 0; i < nmax; i++) { /* for each body */ x[i] = x[i]new ; /* update velocity and position*/ v[i] = v[i]new ;

2.3 N-Body Problem Parallel Code
The algorithm is an O(N2 ) algorithm (for one iteration) as each of the N bodies is influenced by each of the other N - 1 bodies. Not feasible to use this direct algorithm for most interesting N-body problems where N is very large.

2.3 N-Body Problem Time complexity can be reduced using the observation that a cluster of distant bodies can be approximated as a single distant body of the total mass of the cluster sited at the center of mass of the cluster:

2.3 N-Body Problem Barnes-Hut Algorithm
Start with whole space in which one cube contains the bodies (or particles). First, this cube is divided into eight subcubes. If a subcube contains no particles, the subcube is deleted from further consideration. If a subcube contains more than one body, it is recursively divided until every subcube contains one body.

2.3 N-Body Problem This process creates an octtree; that is, a tree with up to eight edges from each node. The leaves represent cells each containing one body. After the tree has been constructed, the total mass and center of mass of the subcube is stored at each node.

2.3 N-Body Problem The force on each body can then be obtained by traversing the tree starting at the root, stopping at a node when the clustering approximation can be used, e.g. when: r ³ d/q where q is a constant typically 1.0 or less (q is called the opening angle). Constructing the tree requires a time of O(n log n), and so does computing all the forces, so that the overall time complexity of the method is O(n log n).

2.3 N-Body Problem

2.3 N-Body Problem Orthogonal Recursive Bisection
Consider a two-dimensional square area First, a vertical line is found that divides the area into two areas each with an equal number of bodies. For each area, a horizontal line is found that divides it into two areas each with an equal number of bodies. Repeated until there are as many areas as processors, and then one processor is assigned to each area.

Partitioning and Divide-and-Conquer concepts as the basis for parallel computing techniques Tree Constructions Examples of Partitioning and Divide-and-Conquer problems Bucket sort numerical integration N-body problem

Pipelined Computations
Chapter 5

In this chapter, we present a parallel processing technique called pipelining, which is applicable to a wide range of problems that are partially sequential in nature. Hence, we can use pipelining to parallelize sequential code. Certain requirements are necessary for improved performance, as will be outlined

1. Pipeline Technique In the pipeline technique, the problem is divided into a series of tasks that have to be completed one after the other. In fact, this is the basis of sequential programming. Each task will be executed by a separate process or processor.

1. Pipeline Technique As an example:
We sometimes refer to each pipeline process as a pipeline stage. This parallelism can be viewed as a form of functional decomposition. As an example: Add all the elements of array a to an accumulating sum: for (i = 0; i < n; i++) sum = sum + a[i];

1. Pipeline Technique The loop could be “unfolded” to yield
sum = sum + a[0]; sum = sum + a[1]; sum = sum + a[2]; sum = sum + a[3]; sum = sum + a[4];

1. Pipeline Technique A frequency filter is a more realistic example in which the problem is divided this way. The objective here is to remove specific frequencies (say the frequencies f0, f1, f2, f3, etc.) from a (digitized) signal, f(t). The signal could enter the pipeline from the left:

1. Pipeline Technique Given that the problem can be divided into a series of sequential tasks, the pipelined approach can provide increased speed under the following three types of computations: If more than one instance of the complete problem is to be executed If a series of data items must be processed, each requiring multiple operations If information to start the next process can be passed forward before the process has completed all its internal operations

1. Pipeline Technique Type 1
Utilized in the internal hardware design of computers. Also appears in simulation exercises where many simulation runs must be completed with different parameters to obtain the comparative results. A Type 1 pipeline can be illustrated in a space-time diagram Two of the major formats of space-time diagrams follow on the next two slides.

1. Pipeline Technique “Type 1” Pipeline Space-Time Diagram

1. Pipeline Technique Alternative Space-Time Diagram

A series of data items must be processed in a sequence Appears in arithmetic calculations such as in multiplying elements of an array

1. Pipeline Technique “Type 2” Pipeline Space-Time Diagram

There is only one instance of the problem to execute, but each process can pass on information to the next process before it has completed.

1. Pipeline Technique If the group of stages is larger than the number of processors in any pipeline, a group of stages can be assigned to each processor, as shown here. Of course, now the pipeline stages within one processor are executed sequentially.

2. Computing Platform for Pipelined Applications.
A key requirement for pipelining is the ability to send messages between adjacent processors in the pipeline. This suggests direct communication links (a line or ring structure) [Transputers]

3. Pipeline Program Examples
Now we are going to look at examples that fit in the three types 1) Adding Numbers 2)Sorting Numbers 3) Prime Number Generation 4) Solving a System of Linear Equations (this is a special case of type 3)

3.1 Adding Numbers For our first example, consider the problem of adding a list of numbers (or any associative operation on a sequence of numbers) The basic code for most processors is recv(&accumulation, Pi-1); accumulation = accumulation + number; send(&accumulation, Pi+1);

3.1 Adding Numbers An SPMD program could have the form:
if (process > 0) { recv(&accumulation, Pi-1); accumulation = accumulation + number; } if (process < n-1) send(&accumulation, Pi+1);

3.1 Adding Numbers The final result is typically in the last process, but it could just return the answer to the master.

3.1 Adding Numbers This structure would also be appropriate for a master/slave setup. We will see this type of setup in Chapter 10 (along with 2D pipelines, and other forms)

3.1 Adding Numbers Coming back to our problem. It does not make sense to have 1 process for each number to add, say, 1000 numbers. Generally, a group of numbers would be added together in each process and the result passed onward. Such data partitioning is used in most numeric computations to reduce the communication overhead. (and is assumed in all the examples)

3.1 Adding Numbers Analysis This is a Type 1 problem.
It is efficient only if we have more than one instance. ttotal = (time for one pipeline cycle) *(number of cycles) ttotal = (tcomp + tcomm)(m+p-1) This assumes there are m instances of the problem. The average time per instance ta = (ttotal)/m

3.1 Adding Numbers Analysis Single Instance of Problem tcomp = 1
tcomm = 2(tstartup + tdata) ttotal = (2(tstartup + tdata) + 1)n Time complexity = O(n).

3.1 Adding Numbers Analysis Multiple Instances of Problem
ttotal = (2(tstartup + tdata) + 1)(m + n - 1) ta = ttotal/m (approx=) 2(tstartup + tdata) + 1 That is, one pipeline cycle

3.1 Adding Numbers Analysis
Data Partitioning with Multiple Instances of Problem. tcomp = d tcomm = 2(tstartup + tdata) ttotal = (2(tstartup + tdata) + d)(m + n/d - 1) As we increase the d, the data partition, the impact of the communication diminishes. But increasing the data partition decreases the parallelism and often increases the execution time.

3.2 Sorting Numbers This is a parallel version of Insertion sort. (See Cormen et al. for the sequential version)

3.2 Sorting Numbers The basic algorithm for process Pi is
recv(&number, Pi-1); if (number > x) { send(&x, Pi+1); x = number; } else send(&number, Pi+1); With n numbers, how many the ith process is to accept is known; it is given by n - i. How many to pass onward is also known; it is given by n - i - 1 since one of the numbers received is not passed onward. Hence, a simple loop could be used.

3.2 Sorting Numbers The pipeline is illustrated here:
A message-passing program using SPMD or a master-slave approach is straightforward

3.2 Sorting Numbers Results of this sorting algorithm can be extracted from the pipeline using either a ring (Fig 5.11) or a bi-directional line configuration

3.2 Sorting Numbers Incorporating results being returned, process i could have the form: right_procno = n - i - 1; /*no of processes to the right */ recv(&x, Pi-1); for (j = 0; j < right_procno; j++) { recv(&number, Pi-1); if (number > x) { send(&x, Pi+1); x = number; } else send(&number, Pi+1); } send(&number, Pi-1); /* send number held */ for (j = 0; j < right_procno; j++) {/*pass on other nos */ recv(&x, Pi+1); send(&x, Pi-1);

3.2 Sorting Numbers Analysis
Assuming that a compare-and-exchange operation is regarded as one computational step, a sequential implementation requires: ts – (n-1)+(n-2)+…+2+1 = ((n*(n-1))/2 This is approximately n2/2 which is obviously a poor sequential sorting algorithm (basically bubble sort) The parallel implementation has n+n-1 = 2n-1 pipeline cycles in the sorting phase. tcomp = 1 tcomm = 2(tstartup + tdata) ttotal = (1+2(tstartup + tdata))(2n-1)

3.2 Sorting Numbers Insertion sort with results returned

3.3 Prime Number Generation
Sieve of Eratosthenes A series of all integers is generated from 2. The first number, 2, is prime and kept. All multiples of this number are deleted as they cannot be prime. The process is repeated with each remaining number. The algorithm removes non-primes, leaving only primes.

Example Suppose we want the prime numbers from 2 to 20. We start with all the numbers: 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20 After considering 2, we get where numbers in red are not prime and not considered further. After considering 3: Subsequent numbers are considered in a similar fashion. To find the primes up to n, it is only necessary to start at numbers up to sqrt(n) . All multiples of numbers greater than sqrt(n) will have been removed as they are also a multiple of some number equal or less than sqrt(n).

Sequential Code Usually employs an array with elements initialized to 1 (TRUE) and set to 0 (FALSE) when the index of the element is not a prime number. Letting the last number be n and the square root of n be sqrt_n, we might have for (i = 2; i < n; i++) prime[i] = 1; /* Initialize array */ for (i = 2; i <= sqrt_n; i++)/* for each number */ if (prime[i] == 1) /* identified as prime */ for (j = i + i; j < n; j = j + i)/*strike multiples */ prime[j] = 0; /* includes already done */ The elements in the array still set to 1 identify the primes (given by the array indices). Then a simple loop accessing the array can find the primes.

Analysis of Sequential Code: The number of iterations striking out multiples of primes will depend upon the prime. There are [n/2 – 1] multiples of 2, [n/3 – 1] multiples of 3, and so on. Hence, the total sequential time is given by ts = [n/2 – 1]+[n/3 - 1]+[n/5 -1]+…+[n/sqrt(n) -1] assuming the computation in each iteration equates to one computational step. The sequential time complexity is O(n2).

Pipelined Implementation:

The code for a process, Pi, could be based upon recv(&x, Pi-1); /* repeat following for each number */ recv(&number, Pi-1); if ((number % x) != 0) send(&number, Pi+1);

Each process will not receive the same amount of numbers and the amount is not known beforehand. Use a “terminator” message, which is sent at the end of the sequence: recv(&x, Pi-1); for (i = 0; i < n; i++) { recv(&number, Pi-1); if (number == terminator) break; if (number % x) != 0) send(&number, Pi+1); }

3.4 Solving a System of Linear Equations – Special Case
Type 3 example - process can continue with useful work after passing on information. To solve system of linear equations of the so-called upper-triangular form: an-1,0x0 + an-1,1x1 + an-1,2x2 … + an-1,n-1xn-1 = bn-1 . a2,0x0 + a2,1x1 + a2,2x2 = b2 a1,0x0 + a1,1x1 = b1 a0,0x = b0 where the a’s and b’s are constants and the x’s are unknowns to be found.

Back Substitution. First, the unknown x0 is found from the last equation; i.e., x0 = b0/a0,0 The value obtained for x0 is substituted into the next equation to obtain x1; i.e., x1 = (b1 – a0,0x0)/a1,1 The values obtained for x1 and x0 are substituted into the next equation to obtain x2: x2 = (b2 – a2,0x0-a2,1x1)/a2,2 and so on until all the unknowns are found.

First pipeline stage computes x0 passes x0 onto the second stage, which computes x1 from x0 and passes both x0 and x1 onto the next stage, which computes x2 from x0 and x1, and so on.

Sequential Code Given the constants ai,j and bk stored in arrays a[][] and b[], respectively, and the values for unknowns to be stored in an array, x[], the sequential code could be: x[0] = b[0]/a[0][0]; /* x[0] computed separately */ for (i = 1; i < n; i++) { /* for remaining unknowns */ sum = 0; for (j = 0; j < i; j++) sum = sum + a[i][j]*x[j]; x[i] = (b[i] - sum)/a[i][i]; }

Parallel Code The pseudocode of process Pi (1 < i < n) of one pipelined version could be for (j = 0; j < i; j++) { recv(&x[j], Pi-1); send(&x[j], Pi+1); } sum = 0; for (j = 0; j < i; j++) sum = sum + a[i][j]*x[j]; x[i] = (b[i] - sum)/a[i][i]; send(&x[i], Pi+1); Now we have additional computations to do after receiving and resending values.

Analysis Cannot assume that the computational effort at each pipeline stage is the same. The first process, P0, performs one divide and one send(). The ith process (0 < i < n - 1) performs i recv()s, i send()s, i multiply/add, one divide/ subtract, and a final send(), a total of 2i + 1 communication times and 2i + 2 computational steps assuming that multiply, add, divide, and subtract are each one step. The last process, Pn-1, performs n - 1 recv()s, n - 1 multiply/adds, and one divide/ subtract, a total of n - 1 communication times and 2n - 1 computational steps.

4. Summary This chapter introduced the following:
The pipeline concept and its application areas Analysis of pipelines Examples illustrating the potential of pipelining Insertion sort Prime number generation Solving an upper triangular system of linear equations.

Further Reading Pipeline processing most often is seen in specialized VLSI components. Systolic Arrays (discussed in Chapter 10) are a special 2D pipeline.

Synchronous Computations
Chapter 6

In this chapter, we will consider problems solved by a group of separate computations that must at times wait for each other before proceeding. A very important class of such applications is called fully synchronous applications. All processes are synchronized at regular points. (Similar to SIMD structure) First we will consider synchronizing processes and then fully synchronous applications.

1. Synchronization 1.1 Barrier
A basic mechanism for synchronizing processes - inserted at the point in each process where it must wait. All processes can continue from this point when all the processes have reached it (or, in some implementations, when a stated number of processes have reached this point).

1.1 Barrier

1.1 Barrier Barriers apply to both shared memory and message-passing systems. In message-passing systems barriers are often provided with library routines.

1.1 Barrier MPI MPI_Barrier() - barrier with a named communicator being the only parameter. Called by each process in the group, blocking until all members of the group have reached the barrier call and only returning then.

1.1 Barrier As usual, MPI does not specify the internal implementation. However, we need to know something about the implementation to assess the complexity of the barrier. So, let us review some of the common implementations of a barrier.

1.2 Counter Implementation
Sometimes called a Linear Barrier

Counter-based barriers often have two phases: A process enters arrival phase and does not leave this phase until all processes have arrived in this phase. Then processes move to departure phase and are released.

Good implementations of a barrier must take into account that a barrier might be used more than once in a process. It might be possible for a process to enter the barrier for a second time before previous processes have left the barrier for the first time. The two phases above handle this scenario.

Example code: Master: for (i = 0; i < n; i++) /*count slaves as they reach barrier*/ recv(P any ); /* release slaves */ send(P i ); Slave processes: send(P master ); recv(P master );

The complete arrangement is illustrated here

1.3 Tree Implementation More efficient.
O(log p) steps Suppose there are eight processes, P 0 , P 1 , P 2 , P 3 , P 4 , P 5 , P 6 , and P 7 : First stage: P 1 sends message to P 0 ; (when P 1 reaches its barrier) P 3 sends message to P 2 ; (when P 3 reaches its barrier) P 5 sends message to P 4 ; (when P 5 reaches its barrier) P 7 sends message to P 6 ; (when P 7 reaches its barrier)

1.3 Tree Implementation Second stage: Third stage:
P 2 sends message to P 0 ; (P 2 and P 3 have reached their barrier) P 6 sends message to P 4 ; (P 6 and P 7 have reached their barrier) Third stage: P 4 sends message to P 0 ; (P 4 , P 5 , P 6 , and P 7 have reached their barrier) P 0 terminates arrival phase; (when P 0 reaches barrier and has received message from P 4 ) Release with a reverse tree construction.

1.3 Tree Implementation

1.4 Butterfly Barrier First stage Second stage Third stage
P 0 « P 1, P 2 « P 3, P 4 « P 5, P 6 « P 7 Second stage P 0 « P 2, P 1 « P 3, P 4 « P 6, P 5 « P 7 Third stage P 0 « P 4, P 1 « P 5, P 2 « P 6, P 3 « P 7

1.5 Local Synchronization
Suppose a process Pi needs to be synchronized and to exchange data with process Pi-1 and process Pi+1 before continuing: Process Pi-1 Process Pi Process Pi+1 recv(P i ); send(P i-1 ); recv(P i ); send(P i ); send(P i+1 ); send(P i ); recv(P i-1 ); recv(P i+1 ); Not a perfect three-process barrier because process Pi-1 will only synchronize with Pi and continue as soon as Pi allows. Similarly, process Pi+1 only synchronizes with Pi.

1.6 Deadlock When a pair of processes each send and receive from each other, deadlock may occur. Deadlock will occur if both processes perform the send, using synchronous routines first (or blocking routines without sufficient buffering). This is because neither will return; they will wait for matching receives that are never reached.

1.6 Deadlock A Solution Example
Arrange for one process to receive first and then send and the other process to send first and then receive. Example Linear pipeline, deadlock can be avoided by arranging so the even-numbered processes perform their sends first and the odd-numbered processes perform their receives first.

1.6 Deadlock Combined deadlock-free blocking sendrecv() routines
MPI provides routine MPI_Sendrecv() and MPI_Sendrecv_replace(). Example Process Pi-1 Process Pi Process Pi+1 sendrecv(P i ); sendrecv(P i-1 ); sendrecv(P i+1 ); sendrecv(P i ); Please note that MPI_Sendrecv has 12 parameters essentially a concatenation of the parameter list for MPI_Send() and MPI_Recv()

2. Synchronized Computations
Can be classififed as: Fully synchronous or Locally synchronous In fully synchronous, all processes involved in the computation must be synchronized. In locally synchronous, processes only need to synchronize with a set of logically nearby processes, not all processes involved in the computation

2.1 Data Parallel Computations
Fully Synchronous Same operation performed on different data elements simultaneously; i.e., in parallel. Particularly convenient because: Ease of programming (essentially only one program). Can scale easily to larger problem sizes. Many numeric and some non-numeric problems can be cast in a data parallel form.

Example To add the same constant to each element of an array: for (i = 0; i < n; i++) a[i] = a[i] + k; The statement a[i] = a[i] + k could be executed simultaneously by multiple processors, each using a different index i (0 < i £ n).

Forall construct Special “parallel” construct in parallel programming languages to specify data parallel operations Example forall (i = 0; i < n; i++) { body } states that n instances of the statements of the body can be executed simultaneously. One value of the loop variable i is valid in each instance of the body, the first instance has i = 0, the next i = 1, and so on. (Think of this as loop unrolling across processors)

To add k to each element of an array, a, we can write forall (i = 0; i < n; i++) a[i] = a[i] + k; Data parallel technique applied to multiprocessors and multicomputers - Example: To add k to the elements of an array: i = myrank; a[i] = a[i] + k;/* body */ barrier(mygroup); where myrank is a process rank between 0 and n-1.

Prefix Sum Problem Given a list of numbers, x0, …, xn-1 , compute all the partial summations (i.e., x0+x1; x0+x1+x2; x0+x1+x2+x3; … ). Can also be defined with associative operations other than addition. Widely studied. Practical applications in areas such as processor allocation, data compaction, sorting, and polynomial evaluation.

The sequential code for the prefix problem could be for(i = 0; i < n; i++) { sum[i] = 0; for (j = 0; j <= i; j++) sum[i] = sum[i] + x[j]; } This is an O(n2) algorithm.

Data parallel method of adding all partial sums of 16 numbers Sequential code for (j = 0; j < log(n); j++)/* at each step */ for (i = 2j; i < n; i++)/*add to accum. sum */ x[i] = x[i] + x[i - 2j]; Written up by Hillis and Steel (1986) requires a different number of computations at each step.

Because SIMD machines must send the same instruction to all processors, you could rewrite it as: Parallel code: for (j = 0; j < log(n); j++)/* at each step */ forall (i = 0; i < n; i++)/* add to accum. sum */ if (i >= 2j) x[i] = x[i] + x[i - 2j]; This uses a maximum of n-1 processors. The time complexity is O(n log n) which is not cost optimal since the sequential was O(n2)

2.2 Synchronous Iteration
Each iteration is composed of several processes they start together at beginning of iteration and the next iteration cannot begin until all processes have finished previous iteration. The forall construct could be used to specify the parallel bodies of the synchronous iteration: for (j = 0; j < n; j++) /*for each synch. iteration */ forall (i = 0; i < N; i++) { /*N procs each executing */ body(i); /*using specific value of i */ }

2.2 Synchronous Iteration
Or it could be written this way: for (j = 0; j<n; j++) {/*for each synchr.iteration*/ i = myrank; /*find value of i to be used */ body(i); /*using specific value of i */ barrier(mygroup); } Let us look at some specific synchronous iteration examples

3. Synchronous Iteration Program Examples
3.1 Solving a System of Linear Equations by Iteration Suppose the equations are of a general form with n equations and n unknowns where the unknowns are x0, x1, x2, … xn-1 (0 £ i < n). an-1,0x0 + an-1,1x1 + a n-1,2 x 2 …+ a n-1,n-1 x n-1 = b n-1 . a2,0x0 + a2,1x1+ a2,2x2 … + a2,n-1xn-1 = b2 a1,0x0 + a1,1x1+ a1,2x2 … + a1,n-1xn-1 = b1 a0,0x0 + a0,1x1+ a0,2x2 … + a0,n-1xn-1 = b0

3.1 Solving a System of Linear Equations by Iteration
One way to solve these equations for the unknowns is by iteration. By rearranging the ith equation: ai,0x0 + ai,1x1+ a2,2x2 … + a2,n-1xn-1 = b2 to xi=(1/ai,i)[bi-(ai,0x 0+ai,1x1+ai,2x2 … ai,i -1xi-1+ai ,i+1xi+1 … +ai,n-1xn-1)] or This equation gives xi in terms of the other unknowns and can be be used as an iteration formula for each of the unknowns to obtain better approximations.

Jacobi Iteration Iterative method described called a Jacobi iteration all values of x are updated together. It can be proven that the Jacobi method will converge if the diagonal values of a have an absolute value greater than the sum of the absolute values of the other a’s on the row(the array of a’s is diagonally dominant) i.e. If This condition is a sufficient but not a necessary condition.

Termination A simple, common approach is to compare values computed in each iteration to the values obtained from the previous iteration, and then to terminate the computation in the tth iteration when all values are within a given tolerance; i.e., when for all i, where xit is the value of xi after the tth iteration and is the value of xit-1 after the (t - 1)th iteration. However, this does not guarantee the solution to that accuracy.

Sequential Code Given arrays a[ ][ ] and b[ ] holding constants in the equations, x[ ] holding unknowns, and fixed number of iterations: for (i = 0; i < n; i++) x[i] = b[i]; /*initialize unknowns*/ for (iteration = 0; iteration < limit; iteration++) { for (i = 0; i < n; i++) { /* for each unknown */ sum = 0; for (j = 0; j < n; j++) /* summation of a[][]x[] */ if (i != j) sum = sum + a[i][j] * x[j]; new_x[i] = (b[i] - sum) / a[i][i];/*compute unknown*/ } x[i] = new_x[i]; /* update values */

Slightly more efficient sequential code: for (i = 0; i < n; i++) x[i] = b[i]; /*initialize unknowns*/ for (iteration = 0; iteration < limit; iteration++) { for (i = 0; i < n; i++) { /* for each unknown */ sum = -a[i][i] * x[i]; for (j = 0; j < n; j++) /* compute summation */ sum = sum + a[i][j] * x[j]; new_x[i] = (b[i] - sum) / a[i][i];/*compute unknown*/ } for (i = 0; i < n; i++) x[i] = new_x[i]; /*update */

Parallel Code - Process P i could be of the form x[i] = b[i]; /*initialize unknown*/ for (iteration = 0; iteration < limit; iteration++) { sum = -a[i][i] * x[i]; for (j = 0; j < n; j++) /* compute summation */ sum = sum + a[i][j] * x[j]; new_x[i] = (b[i] - sum) / a[i][i];/* compute unknown */ broadcast_receive(&new_x[i]); /* broadcast value */ global_barrier(); /* wait for all processes */ } broadcast_receive(), sends the newly computed value of x[i] from process i to every other process and collects data broadcast from the other processes. An alternative simple solution is to return to basic send()s and recv()s.

Allgather Broadcast and gather values in one composite construction.

Iterate until approximations are sufficiently close: x[i] = b[i]; /*initialize unknown*/ iteration = 0; do { iteration++; sum = -a[i][i] * x[i]; for (j = 1; j < n; j++) /* compute summation */ sum = sum + a[i][j] * x[j]; new_x[i] = (b[i] - sum) / a[i][i];/* compute unknown */ broadcast_receive(&new_x[i]); /*broadcast value & wait */ } while (tolerance() && (iteration < limit)); where tolerance() returns FALSE if ready to terminate; otherwise it returns TRUE.

Partitioning Usually the number of processors much fewer than number of data items to be processed. Partition the problem so that processors take on more than one data item.

In the problem at hand, each process can be responsible for computing a group of unknowns. block allocation – allocate groups of consecutive unknowns to processors in increasing order. cyclic allocation – processors are allocated one unknown in order; i.e., processor P0 is allocated x0, xp , x2p , …, x((n/p)-1)p , processor P1 is allocated x1, xp+1, x2p+1, …, x((n/p)1)p+1, and so on. Cyclic allocation no particular advantage here (Indeed, may be disadvantageous because the indices of unknowns have to be computed in a more complex way).

Analysis Suppose there are n equations and p processors. A processor operates upon n/p unknowns. Suppose there are t iterations. One iteration has a computational phase and a broadcast communication phase. Computation. tcomp = n/p(2n + 4)t Communication. tcomm = p(tstartup+(n/p)tdata)t = (ptstartup+ntdata)t Overall. tp = (n/p(2n + 4) + ptstartup+ ntdata)t The resulting total execution time has a minimum value.

Effects of computation and communication in Jacobi iteration

3.2 Heat Distribution Problem
Locally Synchronous Computation A square metal sheet has known temperatures along each of its edges. Find the temperature distribution.

Dividing the area into a fine mesh of points, hi,j. The temperature at an inside point can be taken to be the average of the temperatures of the four neighboring points. Convenient to describe the edges by points adjacent to the interior points. The interior points of hi,j are where 0<i< n, 0< j< n [(n - 1) ´ (n - 1) interior points].

Temperature of each point by iterating the equation: (0 < i < n, 0 < j < n) for a fixed number of iterations or until the difference between iterations of a point is less than some very small prescribed amount.

Natural Ordering

Points numbered from 1 for convenience and include those representing the edges. Each point will then use the equation Could be written as a linear equation containing the unknowns x i-k , x i-1 , x i+1 , and x i+k : xi-k+xi-1-4x i+xi+1+xi-k= 0 Known as the finite difference method. Also solving Laplace’s equation.

Sequential Code Using a fixed number of iterations for (iteration = 0; iteration < limit; iteration++) { for (i = 1; i < n; i++) for (j = 1; j < n; j++) g[i][j] = 0.25*(h[i-1][j]+h[i+1][j]+h[i][j-1]+h[i][j+1]); for (i = 1; i < n; i++) /* update points */ h[i][j] = g[i][j]; }

To stop at some precision: do { for (i = 1; i < n; i++) for (j = 1; j < n; j++) g[i][j] = 0.25*(h[i-1][j]+h[i+1][j]+h[i][j-1]+h[i][j+1]); for (i = 1; i < n; i++)/* update points */ h[i][j] = g[i][j]; continue = FALSE; /* indicates whether to continue */ for (i = 1; i < n; i++)/* check each pt for convergence */ if (!converged(i,j) {/* point found not converged */ continue = TRUE; break; } } while (continue == TRUE);

Parallel Code Version with a fixed number of iterations, process P i,j (except for the boundary points): for (iteration = 0; iteration < limit; iteration++) { g = 0.25 * (w + x + y + z); send(&g, P i-1,j ); /* non-blocking sends */ send(&g, P i+1,j ); send(&g, P i,j-1 ); send(&g, P i,j+1 ); recv(&w, P i-1,j ); /* synchronous receives */ recv(&x, P i+1,j ); recv(&y, P i,j-1 ); recv(&z, P i,j+1 ); }

It is important to use send()s that do not block while waiting for the recv()s; Otherwise the processes would deadlock, each waiting for a recv() before moving on recv()s must be synchronous and wait for the send()s. Each process is synchronized with its four neighbors by the recv()s.

Version where processes stop when they reach their required precision: iteration = 0; do { iteration++; g = 0.25 * (w + x + y + z); send(&g, P i-1,j ); /* locally blocking sends */ send(&g, P i+1,j ); send(&g, P i,j-1 ); send(&g, P i,j+1 ); recv(&w, P i-1,j ); /* locally blocking receives */ recv(&x, P i+1,j ); recv(&y, P i,j-1 ); recv(&z, P i,j+1 ); } while((!converged(i, j)) || (iteration < limit)); send(&g, &i, &j, &iteration, P master );

To handle the edges: if (last_row) w = bottom_value; if (first_row) x = top_value; if (first_column) y = left_value; if (last_column) z = right_value; iteration = 0; do { iteration++; g = 0.25 * (w + x + y + z); if !(first_row) send(&g, P i-1,j ); if !(last_row) send(&g, P i+1,j ); if !(first_column) send(&g, P i,j-1 ); if !(last_column) send(&g, P i,j+1 ); if !(last_row) recv(&w, P i-1,j ); if !(first_row) recv(&x, P i+1,j ); if !(first_column) recv(&y, P i,j-1 ); if !(last_column) recv(&z, P i,j+1 ); } while((!converged) || (iteration < limit)); send(&g, &i, &j, iteration, P master );

Example A room has four walls and a fireplace. Temperature of wall is 20°C, and temperature of fireplace is 100°C. Write a parallel program using Jacobi iteration to compute the temperature inside the room and plot (preferably in color) temperature contours at 10°C intervals

Sample Student Output

Partitioning Normally allocate more than one point to each processor, because there would be many more points than processors. Points could be partitioned into square blocks or strips:

Block partition: Four edges where data points are exchanged. Communication time is given by

Strip partition Two edges where data points are exchanged. Communication time is given by

Optimum In general, the strip partition is best for a large startup time, and a block partition is best for a small startup time. With the previous equations, the block partition has a larger communication time than the strip partition if or (p ³ 9).

Ghost Points An additional row of points at each edge that hold the values from the adjacent edge. Each array of points is increased to accommodate the ghost rows.

Safety and Deadlock When all processes send their messages first and then receive all of their messages, as in all the code so far, it is described as “unsafe” in the MPI literature because it relies upon buffering in the send()s. The amount of buffering is not specified in MPI. If a send routine has insufficient storage available when it is called, the implementation should be such to delay the routine from returning until storage becomes available or until the message can be sent without buffering.

Safety and Deadlock (cont.) Hence, the locally blocking send() could behave as a synchronous send(), only returning when the matching recv() is executed. Since a matching recv() would never be executed if all the send()s are synchronous, deadlock would occur.

Making the code safe Alternate the order of the send()s and recv()s in adjacent processes. This is so that only one process performs the send()s first: The question is how?

Making the code safe (cont.) if ((myid % 2) == 0) { /* even processes */ send(&g[1][1], &m, P i-1 ); recv(&h[1][0], &m, P i-1 ); send(&g[1,m], &m, P i+1 ); recv(&h[1][m+1], &m, P i+1 ); } else { /* odd numbered processes */ }

Making the code safe (cont) Then even synchronous send()s would not cause deadlock. In fact, a good way you can test for safety is to replace message-passing routines in a program with synchronous versions.

MPI Safe message Passing Routines MPI offers several alternative methods for safe communication: Combined send and receive routines: MPI_Sendrecv() (which is guaranteed not to deadlock) Buffered send()s: MPI_Bsend() — here the user provides explicit storage space Nonblocking routines: MPI_Isend() and MPI_Irecv() — here the routine returns immediately, and a separate routine is used to establish whether the message has been received (MPI_Wait(), MPI_Waitall(), MPI_Waitany(), MPI_Test(), MPI_Testall(), or MPI_Testany())

MPI Safe message Passing Routines (cont.) A pseudocode segment using the third method is isend(&g[1][1], &m, P i-1 ); isend(&g[1,m], &m, P i+1 ); irecv(&h[1][0], &m, P i-1 ); irecv(&h[1][m+1], &m, P i+1 ); waitall(4); Wait routine becomes a barrier, waiting for all message-passing routines to complete.

3.3 Cellular Automata In this approach, the problem space is first divided into cells. Each cell can be in one of a finite number of states. Cells are affected by their neighbors according to certain rules, and all cells are affected simultaneously in a “generation.” The rules are reapplied in subsequent generations so that cells evolve, or change state, from generation to generation.

3.3 Cellular Automata The Game of Life
The most famous cellular automata is the “Game of Life” devised by John Horton Conway, a Cambridge mathematician, and published by Gardner (Gardner, 1967). The Game of Life Board game; Board consists of a (theoretically infinite) two-dimensional array of cells. Each cell can hold one “organism” and has eight neighboring cells, including those diagonally adjacent. Initially, some of the cells are occupied in a pattern.

3.3 Cellular Automata The following rules apply:
Every organism with two or three neighboring organisms survives for the next generation. Every organism with four or more neighbors dies from overpopulation. Every organism with one neighbor or none dies from isolation. Each empty cell adjacent to exactly three occupied neighbors will give birth to an organism. These rules were derived by Conway “after a long period of experimentation.”

3.3 Cellular Automata Simple Fun Examples of Cellular Automata
“Sharks and Fishes” An ocean could be modeled as a three-dimensional array of cells. Each cell can hold one fish or one shark (but not both). Sharks and Fishes follow rules

3.3 Cellular Automata Fish Might move around according to these rules:
If there is one empty adjacent cell, the fish moves to this cell. If there is more than one empty adjacent cell, the fish moves to one cell chosen at random. If there are no empty adjacent cells, the fish stays where it is. If the fish moves and has reached its breeding age, it gives birth to a baby fish, which is left in the vacating cell. Fish die after x generations.

3.3 Cellular Automata Sharks Might be governed by the following rules:
If one adjacent cell is occupied by a fish, the shark moves to this cell and eats the fish. If more than one adjacent cell is occupied by a fish, the shark chooses one fish at random, moves to the cell occupied by the fish, and eats the fish. If no fish are in adjacent cells, the shark chooses an unoccupied adjacent cell to move to in a similar manner as fish move. If the shark moves and has reached its breeding age, it gives birth to a baby shark, which is left in the vacating cell. If a shark has not eaten for y generations, it dies.

3.3 Cellular Automata Serious Applications for Cellular Automata
Examples: fluid/gas dynamics the movement of fluids and gases around objects diffusion of gases biological growth airflow across an airplane wing erosion/movement of sand at a beach or riverbank.

4. Partially Synchronous Methods
Computations in which individual processes operate without needing to synchronize with other processes on every iteration. Important idea because synchronizing processes is an expensive operation which very significantly slows the computation and a major cause for reduced performance of parallel programs is due to the use of synchronization. Global synchronization done with barrier routines. Barriers cause processor to wait sometimes needlessly.

The concept of a barrier and its implementations (global barriers and local barriers) Data parallel computations The concept of synchronous iteration Examples using global and local barriers The concept of safe communication Cellular automata

Further Reading Cray – Hardware barrier implementation
Pacheco (97) and Snir (96) develop MPI code for Jacobi iterations Chaotic relaxation offers the potential for very significant improvement in execution speed over fully synchronous methods, when it can be applied

Load Balancing and Termination Detection
Chapter 7

In this chapter, we introduce the concept of load balancing, which is used to distribute computations fairly across processors in order to obtain the highest possible execution speed. A related issue is detecting when a computation has been completed, so-called termination detection. This is more difficult when the computation is distributed.

1. Load Balancing Ideally, we want all the processors to be operating continuously on tasks that would lead to the minimum execution time. Load balancing was mentioned in Chapter 3 for the Mandelbrot calculation, in which there was no interprocess communication. Now we will develop load balancing further to include the case in which there is communication between processors

1. Load Balancing Load balancing is particularly useful when the amount of work is not known prior to execution.

1. Load Balancing Static Load Balancing
Before the execution of any process. Some potential static load-balancing techniques: Round robin algorithm — passes out tasks in sequential order of processes coming back to the first when all processes have been given a task Randomized algorithms — selects processes at random to take tasks Recursive bisection — recursively divides the problem into subproblems of equal computational effort while minimizing message passing Simulated annealing — an optimization technique Genetic algorithm — another optimization technique, described in Chapter 12

1. Load Balancing Several fundamental flaws with static load balancing even if a mathematical solution exists: Very difficult to estimate accurately the execution times of various parts of a program without actually executing the parts. Communication delays that vary under different circumstances Some problems have an indeterminate number of steps to reach their solution.

2. Dynamic Load Balancing
During the execution of the processes. All previous factors are taken into account by making the division of load dependent upon the execution of the parts as they are being executed. Does incur an additional overhead during execution, but it is much more effective than static load balancing

Processes and Processors Computation will be divided into work or tasks to be performed, and processes perform these tasks. Processes are mapped onto processors. Since our objective is to keep the processors busy, we are interested in the activity of the processors. However, we often map a single process onto each processor, so we will use the terms process and processor somewhat interchangeably.

Dynamic load balancing can be classified as one of the following: Centralized Tasks are handed out from a centralized location. A clear master-slave structure exists. Decentralized Tasks are passed between arbitrary processes. A collection of worker processes operate upon the problem and interact among them-selves, finally reporting to a single process. A worker process may receive tasks from other worker processes and may send tasks to other worker processes (to complete or pass on at their discretion).

2.1 Centralized Dynamic Load Balancing
Master process(or) holds the collection of tasks to be performed. Tasks are sent to the slave processes. When a slave process completes one task, it requests another task from the master process. Terms used : work pool, replicated worker, processor farm.

Termination Stopping the computation when the solution has been reached. When tasks are taken from a task queue, computation terminates when: The task queue is empty and Every process has made a request for another task without any new tasks being generated it is not sufficient to terminate when the task queue is empty if one or more processes are still running if a running process may provide new tasks for the task queue. In some applications, a slave may detect the program termination condition by some local termination condition, such as finding the item in a search algorithm.

2.2 Decentralized Dynamic Load Balancing

Fully Distributed Work Pool Processes to execute tasks from each other

Task Transfer Mechanisms Receiver-Initiated Method A process requests tasks from other processes it selects. Typically, a process would request tasks from other processes when it has few or no tasks to perform. Method has been shown to work well at high system load. Unfortunately, it can be expensive to determine process loads.

Sender-Initiated Method A process sends tasks to other processes it selects. Typically, in this method, a process with a heavy load passes out some of its tasks to others that are willing to accept them. Method has been shown to work well for light overall system loads. Another option is to have a mixture of both methods. Unfortunately, it can be expensive to determine process loads. In very heavy system loads, load balancing can also be difficult to achieve because of the lack of available processes.

Process Selection

Process Selection (cont.) Algorithms for selecting a process: Round robin algorithm – process P i requests tasks from process P x , where x is given by a counter that is incremented after each request, using modulo n arithmetic (n processes), excluding x = i. Random polling algorithm – process P i requests tasks from process P x , where x is a number that is selected randomly between 0 and n - 1 (excluding i).

2.3 Load Balancing Using a Line Structure

The master process (P0 in Figure 7.6) feeds the queue with tasks at one end, and the tasks are shifted down the queue. When a “worker” process, Pi (1 £ i < n), detects a task at its input from the queue and the process is idle, it takes the task from the queue. Then the tasks to the left shuffle down the queue so that the space held by the task is filled. A new task is inserted into the left side end of the queue. Eventually, all processes will have a task and the queue is filled with new tasks. High- priority or larger tasks could be placed in the queue first.

Shifting Actions could be orchestrated by using messages between adjacent processes: For left and right communication For the current task

Code Using Time Sharing Between Communication and Computation Master process (P0 ) for (i = 0; i < no_tasks; i++) { recv(P 1 , request_tag); /* request for task */ send(&task, P i , task_tag); /* send tasks into queue */ } send(&empty, P i , task_tag); /* end of tasks */

Process Pi (1 < i < n) if (buffer == empty) { send(P i-1 , request_tag); /* request new task */ recv(&buffer, P i-1 , task_tag); /* task from left proc */ } if ((buffer == full) && (!busy)) {/* get next task */ task = buffer; /* get task*/ buffer = empty; /* set buffer empty */ busy = TRUE; /* set process busy */ nrecv(P i+1 , request_tag, request); /* check msg from right */ if (request && (buffer == full)) { send(&buffer, P i+1 ); /* shift task forward */ buffer = empty; if (busy) { /* continue on current task */ Do some work on task. If task finished, set busy to false. Nonblocking nrecv() is necessary to check for a request being received from the right.

Nonblocking Receive Routines - MPI Nonblocking receive, MPI_Irecv(), returns a request “handle,” which is used in subsequent completion routines to wait for the message or to establish whether the message has actually been received at that point (MPI_Wait() and MPI_Test(), respectively). In effect, the nonblocking receive, MPI_Irecv(), posts a request for message and returns immediately.

Tree Structure Tasks passed from node into one of the two nodes below it when node buffer empty.

3. Distributed Termination Detection Algorithms
3.1 Termination Conditions In general, distributed termination at time t requires the following conditions to be satisfied: Application-specific local termination conditions exist throughout the collection of processes, at time t. There are no messages in transit between processes at time t.

3.1 Termination Condition
Subtle difference between these termination conditions and those given for a centralized load-balancing system is having to take into account messages in transit. The second condition is necessary for the distributed termination system because a message in transit might restart a terminated process. This is more difficult to recognize. The time that it takes for messages to travel between processes will not be known in advance.

3.2 Using Acknowledgment Messages
Each process in one of two states: 1. Inactive - without any task to perform 2. Active The process that sent the task to make it enter the active state becomes its “parent.” On every occasion when process receives a task, it immediately sends an acknowledgment message, except if the process it receives the task from is its parent process.

It only sends an acknowledgment message to its parent when it is ready to become inactive, i.e. When Its local termination condition exists (all tasks are completed). It has transmitted all its acknowledgments for tasks it has received. It has received all its acknowledgments for tasks it has sent out. The last condition means that a process must become inactive before its parent process. When the first process becomes idle, the computation can terminate.

3.3 Ring Termination Algorithms
Single-pass ring termination algorithm 1. When P0 has terminated, it generates a token that is passed to P1 . 2. When Pi (1 £ i < n) receives the token and has already terminated, it passes the token onward to Pi+1. Otherwise, it waits for its local termination condition and then passes the token onward. Pn-1 passes the token to P0 . 3. When P0 receives a token, it knows that all processes in the ring have terminated. A message can then be sent to all processes informing them of global termination, if necessary.

Single-pass ring termination algorithm (cont.) The algorithm assumes that a process cannot be reactivated after reaching its local termination condition. This does not apply to work pool problems in which a process can pass a new task to an idle process

Dual-Pass Ring Termination Algorithm Can handle processes being reactivated but requires two passes around the ring. The reason for reactivation is for process Pi , to pass a task to Pj where j < i and after a token has passed Pj ,. If this occurs, the token must recirculate through the ring a second time. To differentiate these circumstances, tokens are colored white or black. Processes are also colored white or black. Receiving a black token means that global termination may not have occurred and the token must be recirculated around the ring again.

The algorithm is as follows, again starting at P0: 1. P0 becomes white when it has terminated and generates a white token to P1. 2. The token is passed through the ring from one process to the next when each process has terminated, but the color of the token may be changed. If Pi passes a task to Pj where j < i (that is, before this process in the ring), it becomes a black process; otherwise it is a white process.

2. (cont) A black process will color a token black and pass it on. A white process will pass on the token in its original color (either black or white). After Pi has passed on a token, it becomes a white process. Pn-1 passes the token to P0. 3. When P0 receives a black token, it passes on a white token; if it receives a white token, all processes have terminated.

Notice that in both ring algorithms, P0 becomes the central point for global termination. Also, it is assumed that an acknowledge signal is generated to each request.

Tree Algorithm Local actions described in Figure 7.11 can be applied to various structures, notably a tree structure, to indicate that processes up to that point have terminated.

3.4 Fixed Energy Distributed Termination Algorithms
Uses the notation of a fixed quantity within the system, colorfully termed “energy.” The system starts with all the energy being held by one process, the master process. Master process passes out portions of the energy with the tasks to processes making requests for tasks. If these processes receive requests for tasks, the energy is divided further and passed to these processes.

3.4 Fixed Energy Distributed Termination Algorithms
When a process becomes idle, it passes the energy it holds back before requesting a new task. A process will not hand back its energy until all the energy it handed out is returned and combined to the total energy held. When all the energy is returned to the root and the root becomes idle, all the processes must be idle and the computation can terminate. Significant disadvantage - dividing the energy will be of finite precision and adding the partial energies may not equate to the original energy. In addition, one can only divide the energy so far before it becomes essentially zero.

4. Program Example In this section, we will discuss how the various load-balancing strategies can be applied to a representative problem. There are several application areas obvious search and optimization areas. Image processing ray tracing volume rendering

4.1 Shortest Path Problem Finding the shortest distance between two points on a graph. It can be stated as follows: Given a set of interconnected nodes where the links between the nodes are marked with “weights,” find the path from one specific node to another specific node that has the smallest accumulated weights. The interconnected nodes can be described by a graph. The nodes are called vertices, and the links are called edges. If the edges have implied directions (that is, an edge can only be traversed in one direction, the graph is a directed graph.

4.1 Shortest Path Problem Graph could be used to find solution to many different problems; for example, 1. The shortest distance between two towns or other points on a map, where the weights represent distance 2. The quickest route to travel, where the weights represent time (the quickest route may not be the shortest route if different modes of travel are available; for example, flying to certain towns) 3. The least expensive way to travel by air, where the weights represent the cost of the flights between cities (the vertices)

4.1 Shortest Path Problem 4. The best way to climb a mountain given a terrain map with contours 5. The best route through a computer network for minimum message delay (the vertices represent computers, and the weights represent the delay between two computers) 6. The most efficient manufacturing system, where the weights represent hours of work “The best way to climb a mountain” will be used as an example.

4.1 Shortest Path Problem Example: The Best Way to Climb a Mountain

4.1 Shortest Path Problem Weights in graph indicate the amount of effort that would be expended in traversing the route between two connected camp sites. The effort in one direction may be different from the effort in the opposite direction (downhill instead of uphill!). (directed graph)

4.2 Graph Representation Two basic ways that a graph can be represented in a program: 1. Adjacency matrix — a two-dimensional array, a, in which a[i][j] holds the weight associated with the edge between vertex i and vertex j if one exists 2. Adjacency list — for each vertex, a list of vertices directly connected to the vertex by an edge and the corresponding weights associated with the edges Adjacency matrix used for dense graphs. The adjacency list is used for sparse graphs. The difference is based upon space (storage) requirements. Accessing the adjacency list is slower than accessing the adjacency matrix.

4.2 Graph Representation

4.3 Searching a Graph Two well-known single-source shortest-path algorithms: Moore’s single-source shortest-path algorithm (Moore, 1957) Dijkstra’s single-source shortest-path algorithm (Dijkstra, 1959) Moore’s algorithm is chosen because it is more amenable to parallel implementation although it may do more work. The weights must all be positive values for the algorithm to work. (Other algorithms exist that will work with both positive and negative weights.)

4.3 Searching a Graph Moore’s Algorithm
Starting with the source vertex, the basic algorithm implemented when vertex i is being considered as follows. Find the distance to vertex j through vertex i and compare with the current minimum distance to vertex j. Change the minimum distance if the distance through vertex i is shorter. In mathematical notation, if di is the current minimum distance from the source vertex to vertex i and wi,j is the weight of the edge from vertex i to vertex j, we have dj = min(dj, di + wi,j)

4.3 Searching a Graph

4.3 Searching a Graph Data Structures and Code
First-in-first-out vertex queue created to hold a list of vertices to examine. Initially, only source vertex is in queue. Current shortest distance from source vertex to vertex i will be stored in the array dist[i] (1 £ i < n) - n vertices, and vertex 0 is the source vertex. At first, none of these distances known and array elements are initialized to infinity.

4.3 Searching a Graph Suppose w[i][j] holds the weight of the edge from vertex i and vertex j (infinity if no edge). The code could be of the form: newdist_j = dist[i] + w[i][j]; if (newdist_j < dist[j]) dist[j] = newdist_j; When a shorter distance is found to vertex j, vertex j is added to the queue (if not already in the queue), which will cause vertex j to be examined again. This is an important aspect of this algorithm, which is not present in Dijkstra’s algorithm.

4.3 Searching a Graph Stages in Searching a Graph Example
The initial values of the two key data structures are

4.3 Searching a Graph After examining A to B:

4.3 Searching a Graph After examining B to F, E, D, and C:

4.3 Searching a Graph After examining E to F

4.3 Searching a Graph After examining D to E:

4.3 Searching a Graph After examining C to D: No changes. After examining E (again) to F : No more vertices to consider. We have the minimum distance from vertex A to each of the other vertices, including the destination vertex, F. Usually, the actual path is also required in addition to the distance. Then the path needs to be stored as distances are recorded. The path in our case is A ® B ® D ® E ® F.

4.3 Searching a Graph Sequential Code
Specific details of maintaining vertex queue omitted. Let next_vertex() return the next vertex from the vertex queue or no_vertex if none. Assume that adjacency matrix used, named w[][]. while ((i = next_vertex()) != no_vertex) /* while a vertex */ for (j = 1; j < n; j++) /* get next edge */ if (w[i][j] != infinity) { /* if an edge */ newdist_j = dist[i] + w[i][j]; if (newdist_j < dist[j]) { dist[j] = newdist_j; append_queue(j); /*vertex to queue if not there*/ } } /* no more vertices to consider */

4.3 Searching a Graph Parallel Implementations Centralized Work Pool
Centralized work pool holds the vertex queue, vertex_queue[] as tasks. Each slave takes vertices from the vertex queue and returns new vertices. Since the structure holding the graph weights is fixed, this structure could be copied into each slave. Assume a copied adjacency matrix.

4.3 Searching a Graph Master: while (vertex_queue() != empty) { };
recv(P ANY , source = P i ); /* request task from slave */ v = get_vertex_queue(); send(&v, P i ); /* send next vertex and */ send(&dist, &n, P i ); /* current dist array */ . recv(&j, &dist[j], P ANY , source = P i );/*new distance*/ append_queue(j, dist[j]); /* append vertex to queue */ /* and update distance array */ }; send(P i , termination_tag); /* termination message*/

4.3 Searching a Graph Slave (process i):
send(P master ); /* send request for task */ recv(&v, P master , tag); /* get vertex number */ if (tag != termination_tag) { recv(&dist, &n, P master ); /* and dist array */ for (j = 1; j < n; j++) /* get next edge */ if (w[v][j] != infinity) { /* if an edge */ newdist_j = dist[v] + w[v][j]; if (newdist_j < dist[j]) { dist[j] = newdist_j; send(&j, &dist[j], P master );/* add vertex to queue */ } /* send updated distance */ }

4.3 Searching a Graph Decentralized Work Pool
Convenient approach is to assign slave process i to search around vertex i only and for it to have the vertex queue entry for vertex i if this exists in the queue. The array dist[] will also be distributed among the processes so that process i maintains the current minimum distance to vertex i. Process i also stores an adjacency matrix/list for vertex i, for the purpose of identifying the edges from vertex i.

4.3 Searching a Graph Search Algorithm
Search activated by loading source vertex into the appropriate process. Vertex A is the first vertex to search. The process assigned to vertex A is activated. This process will search around its vertex to find distances to connected vertices. Distance to process j will be sent to process j for it to compare with its currently stored value and replace if the currently stored value is larger. In this fashion, all minimum distances will be updated during the search. If the contents of d[i] changes, process i will be reactivated to search again.

4.3 Searching a Graph

4.3 Searching a Graph Slave (process i) dist = newdist;
recv(newdist, PANY); if (newdist < dist) { dist = newdist; vertex_queue = TRUE; /* add to queue */ } else vertex_queue == FALSE; if (vertex_queue == TRUE) /* start searching around vertex */ for (j = 1; j < n; j++) /* get next edge */ if (w[j] != infinity) { d = dist + w[j]; send(&d, P j ); /* send distance to proc j */ }

4.3 Searching a Graph Simplified slave (process i)
recv(newdist, P ANY ); if (newdist < dist) dist = newdist; /* start searching around vertex */ for (j = 1; j < n; j++) /* get next edge */ if (w[j] != infinity) { d = dist + w[j]; send(&d, P j ); /* send distance to proc j */ }

4.3 Searching a Graph Mechanism necessary to repeat the actions and terminate when all processes are idle and must cope with messages in transit. Simplest solution Use synchronous message passing, in which a process cannot proceed until the destination has received the message. Note that a process is only active after its vertex is placed on the queue, and it is possible for many processes to be inactive, leading to an inefficient solution. The method is also impractical for a large graph if one vertex is allocated to each processor. In that case, a group of vertices could be allocated to each processor.

Centralized and distributed work pools and load-balancing techniques Several distributed termination algorithms Shortest-path graph searching applications

Further Reading There have been a large number of research papers on static and dynamic scheduling of tasks over the years. Load balancing in distributed systems is also described in many papers. Termination detection is treated in several texts. Graph Algorithms are covered in several papers and texts over the years.

Programming with Shared Memory
Chapter 8

In this chapter, we outline methods of programming systems having shared memory, notably the use of system/library calls and the use of parallel programming languages. For the system/library call approach, we describe both using standard UNIX processes and using threads (concentrating on Pthreads) For the parallel programming language approach, we limit the discussion to general features and techniques.

1. Shared Memory Multiprocessors
Any memory location can be accessible by any of the processors. A single address space exists, meaning that each memory location is given a unique address within a single range of addresses.

For small number of processors, common architecture is the single bus architecture: This architecture is only suitable for, perhaps, up to eight processors, because the bus can only be used by one processor at a time.

An alternative architecture:

Several Alternatives for Programming: Using a new programming language Using library routines with an existing sequential language Using a sequential programming language and ask a parallelizing compiler to convert it into parallel executable code. UNIX Processes Threads (Pthreads, Java, ..)

Several Alternatives for Programming (cont): Modifying the syntax of an existing sequential programming language to create a parallel programming language. Example UPC Using an existing sequential programming language supplemented with compiler directives for specifying parallelism. Example OpenMP

2. Constructs for Specifying Parallelism
2.1. Creating Concurrent Processes FORK-JOIN was described by Conway in 1963, and was known before 1960. In the original construct a FORK statement generates one new path for a concurrent process and the concurrent processes use the JOIN statement at their ends.

2.1. Creating Concurrent Processes
UNIX Heavyweight Processes The UNIX system call fork() creates a new process. The new process (child process) is an exact copy of the calling process except that it has a unique process ID. It has its own copy of the parent’s variables. They are assigned the same values as the original variables initially. The forked process starts execution at the point of the fork. On success, fork() returns 0 to the child process and returns the process ID of the child process to the parent process.

Processes are “joined” with the system calls wait() and exit() defined as wait(statusp); /*delays caller until signal received or one of its */ /*child processes terminates or stops */ exit(status); /*terminates a process. */ A single child process can be created by pid = fork(); /* fork */ Code to be executed by both child and parent if (pid == 0) exit(0); else wait(0); /* join */

If the child is to execute different code, could use pid = fork(); if (pid == 0) { code to be executed by slave } else { Code to be executed by parent } if (pid == 0) exit(0); else wait(0);

2.2 Threads “heavyweight” process -completely separate program with its own variables, stack, and memory allocation. Threads - shares the same memory space and global variables between routines.

2.2 Threads IEEE Portable Operating System Interface, POSIX, section standard

2.2 Threads Pthread Barrier
The routine pthread_join() waits for one specific thread to terminate. To create a barrier waiting for all threads, pthread_join() could be repeated: for (i = 0; i < n; i++) pthread_create(&thread[i], NULL, (void *) slave, (void *) &arg); pthread_join(thread[i], NULL);

2.2 Threads Detached Threads
It may be that a thread may not be bothered when a thread it creates terminates and in that case a join not be needed. Threads that are not joined are called detached threads.

2.2 Threads Thread Pools All of the configurations that have been described for processes are applicable to threads. A master thread could control a collection of slave threads. A work pool of threads could be formed. Threads cans communicate through shared locations or, as we shall see, using signals

2.2 Threads Statement Execution Order
On a multiprocessor system, instructions of individual processes/threads might be interleaved in time. Example Process 1 Process 2 Instruction 1.1 Instruction 2.1 Instruction 1.2 Instruction 2.2 Instruction 1.3 Instruction 2.3

2.2 Threads there are several possible orderings, including:
Instruction 1.1 Instruction 1.2 Instruction 2.1 Instruction 1.3 Instruction 2.2 Instruction 2.3 assuming an instruction cannot be divided into smaller interruptible steps.

2.2 Threads If two processes were to print messages, for example, the messages could appear in different orders depending upon the scheduling of processes calling the print routine. Worse, the individual characters of each message could be interleaved if the machine instructions of instances of the print routine could be interleaved.

2.2 Threads Compiler/Processor Optimations
Compiler (or processor) might reorder instructions for optimization purposes. Example The statements a = b + 5; x = y + 4; could be compiled to execute in reverse order: and still be logically correct.

2.2 Threads It may be advantageous to delay statement a = b + 5
because some previous instruction currently being executed in the processor needs more time to produce the value for b. It is very common for modern processors to execute machines instructions out of order for increased speed of execution.

2.2 Threads Thread-Safe Routines
System calls or library routines are called thread safe if they can be called from multiple threads simultaneously and always produce correct results. Example Standard I/O thread safe (prints messages without interleaving the characters). Routines that access shared/static data may require special care to be made thread safe.

2.2 Threads Example System routines that return time may not be thread safe. The thread-safety aspect of any routine can be avoided by forcing only one thread to execute the routine at a time. This could be achieved by simply enclosing the routine in a critical section but this is very inefficient.

3. Sharing Data The key aspect of shared memory programming is that shared memory provides the possibility of creating variables and data structures that can be accessed directly by every processor rather than having to pass data in messages.

3.1 Creating Shared Data If UNIX heavyweight processes are to share data, additional shared memory system calls are necessary. The shared memory system calls allow processes to attach a segment of physical memory to their virtual memory space. This is created using shmget() Once created it is attached using shmat() It is not necessary to create shared data items explicitly when using threads.

3.2 Accessing Shared Data Consider two processes each of which is to add one to a shared data item, x. Necessary for the contents of the x location to be read, x + 1 computed, and the result written back to the location. With two processes doing this at approximately the same time, we have

3.2 Accessing Shared Data An example given by Nichols, Buttlar, and Farrell (1996) is with two automatic teller machines (one being accessed by the husband and one by the wife simultaneously). This problem can be generalized to shared resources.

3.2 Accessing Shared Data Critical Section
A mechanism for ensuring that only one process accesses a particular resource at a time is to establish sections of code involving the resource as so-called critical sections and arrange that only one such critical section is executed at a time. The first process to reach a critical section for a particular resource enters and executes the critical section. The process prevents all other processes from their critical sections for the same resource. Once the process has finished its critical section, another process is allowed to enter a critical section for the same resource. This mechanism is known as mutual exclusion.

3.2 Accessing Shared Data Locks
The simplest mechanism for ensuring mutual exclusion of critical sections is by the use of a lock. A lock is a 1-bit variable that is a 1 to indicate that a process has entered the critical section and a 0 to indicate that no process is in the critical section. The lock operates much like that of a door lock. A process coming to the “door” of a critical section and finding it open may enter the critical section, locking the door behind it to prevent other processes from entering. Once the process has finished the critical section, it unlocks the door and leaves.

3.2 Accessing Shared Data Spin Lock Example:
while (lock == 1) do_nothing; /* no operation in while loop */ lock = 1; /* enter critical section */ critical section lock = 0; /* leave critical section */

3.2 Accessing Shared Data

3.2 Accessing Shared Data Pthread Lock Routines
Locks are implemented in Pthreads with mutually exclusive lock variables, or “mutex” variables. To use mutex, it must be declared as of type pthread_mutex_t and initialized: pthread_mutex_t mutex1; . pthread_mutex_init(&mutex1, NULL); NULL specifies a default attribute for the mutex. A mutex can be destroyed with pthread_mutex_destroy().

3.2 Accessing Shared Data A critical section can then be protected using pthread_mutex_lock() and pthread_mutex_unlock(): pthread_mutex_lock(&mutex1); . critical section pthread_mutex_unlock(&mutex1); If a thread reaches a mutex lock and finds it locked, it will wait for the lock to open. If more than one thread is waiting for the lock to open when it opens, the system will select one thread to be allowed to proceed. Only the thread that locks a mutex can unlock it.

3.2 Accessing Shared Data Deadlock
Can occur with two processes when one requires a resource held by the other, and this process requires a resource held by the first process.

3.2 Accessing Shared Data Can also occur in a circular fashion with several processes having a resource wanted by another. Deadlock can be eliminated between two processes accessing more than one resource if both processes make requests first for one resource and then for the other.

3.2 Accessing Shared Data Pthreads
Offers one routine that can test whether a lock is actually closed without blocking the thread pthread_mutex_trylock(). This routine will lock an unlocked mutex and return 0 or will return with EBUSY if the mutex is already locked You might find this useful in overcoming deadlock.

3.2 Accessing Shared Data Semaphores
A semaphore, s (say), is a positive integer (including zero) operated upon by two operations named P and V. P operation, P(s) Waits until s is greater than zero and then decrements s by one and allows process to continue. V operation, V(s) increments s by one to release one of the waiting processes (if any).

3.2 Accessing Shared Data Semaphores (cont.)
The P and V operations are performed indivisibly. Mechanism for activating waiting processes is implicit in P and V operations. Though the exact algorithm is not specified, algorithm is expected to be fair. Processes delayed by P(s) are kept in abeyance until released by a V(s) on the same semaphore. So, Why P and V?

3.2 Accessing Shared Data Semaphores were Devised by Dijkstra in 1968.
Letter P is from the Dutch word passeren, meaning “to pass,” and Letter V is from the Dutch word vrijgeven, meaning “to release.”

3.2 Accessing Shared Data Mutual Exclusion of Critical Sections
Can be achieved with one semaphore having the value 0 or 1 (a binary semaphore), which acts as a lock variable, but the P and V operations include a process scheduling mechanism. The semaphore is initialized to 1, indicating that no process is in its critical section associated with the semaphore. Process 1 Process 2 Process 3 Noncritical section Noncritical section Noncritical section . . . P(s) P(s) P(s) Critical section Critical section Critical section V(s) V(s) V(s)

3.2 Accessing Shared Data General Semaphore
Can take on positive values other than zero and one. Such semaphores provide, for example, a means of recording the number of “resource units” available or used and can be used to solve producer/consumer problems. Semaphore routines exist for UNIX processes. They do not exist in Pthreads as such, though they can be written and they do exist in the real-time extension to Pthreads.

3.2 Accessing Shared Data Monitor
A suite of procedures that provides the only method to access a shared resource. Essentially the data and the operations that can operate upon the data are encapsulated into one structure. Reading and writing can only be done by using a monitor procedure, and only one process can use a monitor procedure at any instant.

3.2 Accessing Shared Data Monitor (cont) Java Monitor
A monitor procedure could be implemented using a semaphore to protect its entry; i.e., monitor_proc1() { P(monitor_semaphore); . monitor body V(monitor_semaphore); return; } Java Monitor The concept of a monitor exists in Java.

3.2 Accessing Shared Data Condition Variables
Often, a critical section is to be executed if a specific global condition exists; for example, if a certain value of a variable has been reached. With locks, the global variable would need to be examined at frequent intervals (“polled”) within a critical section. This is a very time-consuming and unproductive exercise. The problem can be overcome by introducing so-called condition variables.

3.2 Accessing Shared Data Condition Variable Operations
Three operations are defined for a condition variable: Wait(cond_var)— wait for a condition to occur Signal(cond_var)— signal that the condition has occurred Status(cond_var)— return the number of processes waiting for the condition to occur The wait operation will also release a lock or semaphore and can be used to allow another process to alter the condition. When the process calling wait() is finally allowed to proceed, the lock or semaphore is again set.

3.2 Accessing Shared Data Example
Consider one or more processes (or threads) designed to take action when a counter, x, is zero. Another process or thread is responsible for decrementing the counter. The routines could be of the form

3.2 Accessing Shared Data Pthread Condition Variables
Associated with a specific mutex. Given the declarations and initializations: pthread_cond_t cond1; pthread_mutex_t mutex1; pthread_cond_init(&cond1, NULL); pthread_mutex_init(&mutex1, NULL);

3.2 Accessing Shared Data the Pthreads arrangement for signal and wait is as follows: Signals are not remembered - threads must already be waiting for a signal to receive it.

4. Parallel Programming Languages and Constructs

4.2 Language Constructs for Parallelism
Shared Data Shared memory variables might be declared as shared with, say, shared int x; par Construct For specifying concurrent statements: par { S1; S2; . Sn; }

forall Construct To start multiple similar processes together: forall (i = 0; i < n; i++) { S1; S2; . Sm; } which generates n processes each consisting of the statements forming the body of the for loop, S1, S2, …, Sm. Each process uses a different value of i.

Example: forall (i = 0; i < 5; i++) a[i] = 0; clears a[0], a[1], a[2], a[3], and a[4] to zero concurrently.

4.3. Dependency Analysis To identify which processes could be executed together. Example We can see immediately in the code forall (i = 0; i < 5; i++) a[i] = 0; that every instance of the body is independent of the other instances and all instances can be executed simultaneously. However, it may not be that obvious. Need algorithmic way of recognizing dependencies, for a parallelizing compiler.

4.3. Dependency Analysis Bernstein's Conditions
Set of conditions that are sufficient to determine whether two processes can be executed simultaneously. Let us define two sets of memory locations: Ii is the set of memory locations read by process Pi . Oj is the set of memory locations altered by process Pj .

4.3. Dependency Analysis Bernstein's Conditions (cont.)
For two processes P1 and P2 to be executed simultaneously, inputs to process P1 must not be part of outputs of P2 , and inputs of P2 must not be part of outputs of P1 ; i.e., I 1 Ç O 2 = f I 2 Ç O 1 = f where f is an empty set. The set of outputs of each process must also be different; i.e., O 1 Ç O 2 = f If the three conditions are all satisfied, the two processes can be executed concurrently.

4.3. Dependency Analysis Example Suppose the two statements are (in C)
a = x + y; b = x + z; We have I 1 = (x, y) O 1 = (a) I 2 = (x, z) O 2 = (b) and the conditions I 1 Ç O 2 = f I 2 Ç O 1 = f O 1 Ç O 2 = f are satisfied. Hence, the statements a = x + y and b = x + z can be executed simultaneously.

5. OpenMP An accepted standard developed in the late 1990s by a group of industry specialists. Consists of a small set of compiler directives, augmented with a small set of library routines and environment variables using the base language Fortran and C/C++. The compiler directives can specify such things as the par and forall operations described previously. Several OpenMP compilers available.

5. OpenMP For C/C++, the OpenMP directives are contained in #pragma statements. The OpenMP #pragma statements have the format: #pragma omp directive_name ... where omp is an OpenMP keyword. May be additional parameters (clauses) after the directive name for different options. Some directives require code to specified in a structured block (a statement or statements) that follows the directive and then the directive and structured block form a “construct”.

6.1 Shared Data in Systems with Caches
All modern computer systems have cache memory, high-speed memory closely attached to each processor for holding recently referenced data and code. Cache coherence protocols In the update policy, copies of data in all caches are updated at the time one copy is altered. In the invalidate policy, when one copy of data is altered, the same data in any other cache is invalidated (by resetting a valid bit in the cache). These copies are only updated when the associated processor makes reference for it.

False Sharing Different parts of block required by different processors but not same bytes. If one processor writes to one part of the block, copies of the complete block in other caches must be updated or invalidated though the actual data is not shared.

Solution for False Sharing Compiler to alter the layout of the data stored in the main memory, separating data only altered by one processor into different blocks.

6.2 Shared Memory Synchronization
High performance programs should have as few as possible critical sections as their use can serialize the code. Suppose, all processes happen to come to their critical section together. They will execute their critical sections one after the other. In that situation, the execution time becomes almost that of a single processor.

6.2 Shared Memory Synchronization

6.3 Sequential Consistency
Formally defined by Lamport (1979): A multiprocessor is sequentially consistent if the result of any execution is the same as if the operations of all the processors were executed in some sequential order, and the operations of each individual processors occur in this sequence in the order specified by its program. i.e. the overall effect of a parallel program is not changed by any arbitrary interleaving of instruction execution in time.

Relaxing Read/Write Orders Processors may be able to relax the consistency in terms of the order of reads and writes of one processor with respect to those of another processor to obtain higher performance, and add instructions to enforce consistency when needed. Some examples of instructions added are next

Examples: Alpha processors Memory barrier (MB) instruction - waits for all previously issued memory accesses instructions to complete before issuing any new memory operations. Write memory barrier (WMB) instruction - as MB but only on memory write operations, i.e. waits for all previously issued memory write accesses instructions to complete before issuing any new memory write operations - which means memory reads could be issued after a memory write operation but overtake it and complete before the write operation.

Examples: SUN Sparc V9 processors memory barrier (MEMBAR) instruction with four bits for variations Write-to-read bit prevent any reads that follow it being issued before all writes that precede it have completed. Other: Write-to-write, read-to-read, read-to-write. IBM PowerPC processor SYNC instruction - similar to Alpha MB instruction

7. Program Examples To sum the elements of an array, a[1000]:
int sum, a[1000]; sum = 0; for (i = 0; i < 1000; i++) sum = sum + a[i];

7.1 UNIX Processes Calculation will be divided into two parts, one doing even i and one doing odd i; i.e., Process 1 Process 2 sum1 = 0; sum2 = 0; for (i = 0; i < 1000; i = i + 2) for (i = 1; i < 1000; i = i + 2) sum1 = sum1 + a[i]; sum2 = sum2 + a[i]; Each process will add its result (sum1 or sum2) to an accumulating result, sum : sum = sum + sum1; sum = sum + sum2; Sum will need to be shared and protected by a lock.

7.1 UNIX Processes Shared data structure is created:
a[] just needs read access sum needs write access Therefore a critical region – use a binary semaphore

7.1 UNIX Processes #include <sys/types.h>
#include <sys/ipc.h> #include <sys/shm.h> #include <sys/sem.h> #include <stdio.h> #include <errno.h> #define array_size 1000 /* no of elements in shared memory */ extern char *shmat(); void P(int *s); void V(int *s); int main() { int shmid, s, pid; /* shared memory, semaphore, proc id */ char *shm; /*shared mem. addr returned by shmat()*/ int *a, *addr, *sum; /* shared data variables*/ int partial_sum; /* partial sum of each process */ int i; /* initialize semaphore set */ int init_sem_value = 1; s = semget(IPC_PRIVATE, 1, (0600 | IPC_CREAT))

7.1 UNIX Processes if (s == -1) { /* if unsuccessful*/
perror("semget"); exit(1); } if (semctl(s, 0, SETVAL, init_sem_value) < 0) { perror("semctl"); /* create segment*/ shmid = shmget(IPC_PRIVATE,(array_size*sizeof(int)+1), (IPC_CREAT|0600)); if (shmid == -1) { perror("shmget"); /* map segment to process data space */ shm = shmat(shmid, NULL, 0); /* returns address as a character*/ if (shm == (char*)-1) { perror("shmat");

7.1 UNIX Processes addr = (int*)shm; /* starting address */
sum = addr; /* accumulating sum */ addr++; a = addr; /* array of numbers, a[] */ *sum = 0; for (i = 0; i < array_size; i++) /* load array with numbers */ *(a + i) = i+1; pid = fork(); /* create child process */ if (pid == 0) { /* child does this */ partial_sum = 0; for (i = 0; i < array_size; i = i + 2) partial_sum += *(a + i); else { /* parent does this */ for (i = 1; i < array_size; i = i + 2) } P(&s); /* for each process, add partial sum */ *sum += partial_sum; V(&s);

7.1 UNIX Processes } /* end of main */
printf("\nprocess pid = %d, partial sum = %d\n", pid, partial_sum); if (pid == 0) exit(0); /* terminate child proc */ else wait(0); printf("\nThe sum of 1 to %i is %d\n", array_size, *sum); /* remove semaphore */ if (semctl(s, 0, IPC_RMID, 1) == -1) { perror("semctl"); exit(1); } /* remove shared memory */ if (shmctl(shmid, IPC_RMID, NULL) == -1) { perror("shmctl"); } /* end of main */

7.1 UNIX Processes void P(int *s) /* P(s) routine*/ {
struct sembuf sembuffer, *sops; sops = &sembuffer; sops->sem_num = 0; sops->sem_op = -1; sops->sem_flg = 0; if (semop(*s, sops, 1) < 0) { perror("semop"); exit(1); } return;

7.1 UNIX Processes void V(int *s) /* V(s) routine */ {
struct sembuf sembuffer, *sops; sops = &sembuffer; sops->sem_num = 0; sops->sem_op = 1; sops->sem_flg = 0; if (semop(*s, sops, 1) <0) { perror("semop"); exit(1); } return;

7.1 UNIX Processes SAMPLE OUTPUT process pid = 0, partial sum = 250000
The sum of 1 to 1000 is

7.2 Pthreads Example In this example, n threads are created, each taking numbers from the list to add to their sums. When all numbers have been taken, the threads can add their partial results to a shared location sum. The shared location global_index is used by each thread to select the next element of a[]. After index is read, it is incremented in preparation for the next element to be read. The result location is sum, as before, and will also need to be shared and access protected by a lock.

7.2 Pthreads Example

7.2 Pthreads Example #include <stdio.h>
#include <pthread.h> #define array_size 1000 #define no_threads 10 /* shared data */ int a[array_size]; /* array of numbers to sum */ int global_index = 0; /* global index */ int sum = 0; /* final result, also used by slaves */ pthread_mutex_t mutex1; /* mutually exclusive lock variable */

7.2 Pthreads Example void *slave(void *ignored) /* Slave threads */ {
int local_index, partial_sum = 0; do { pthread_mutex_lock(&mutex1);/* get next index into the array */ local_index = global_index;/* read current index & save locally*/ global_index++; /* increment global index */ pthread_mutex_unlock(&mutex1); if (local_index < array_size) partial_sum += *(a + local_index); } while (local_index < array_size); pthread_mutex_lock(&mutex1); /* add partial sum to global sum */ sum += partial_sum; return (); /* Thread exits */ }

7.2 Pthreads Example main () { } /* end of main */ int i;
pthread_t thread[10]; /* threads */ pthread_mutex_init(&mutex1,NULL); /* initialize mutex */ for (i = 0; i < array_size; i++) /* initialize a[] */ a[i] = i+1; for (i = 0; i < no_threads; i++) /* create threads */ if (pthread_create(&thread[i], NULL, slave, NULL) != 0) perror("Pthread_create fails"); for (i = 0; i < no_threads; i++) /* join threads */ if (pthread_join(thread[i], NULL) != 0) perror("Pthread_join fails"); printf("The sum of 1 to %i is %d\n", array_size, sum); } /* end of main */

7.2 Pthreads Example SAMPLE OUTPUT The sum of 1 to 1000 is

7.3 Java Example: public class Adder { public int[ ] array;
private int sum = 0; private int index = 0; private int number_of_threads = 10; private int threads_quit; public Adder() threads_quit = 0; array = new int[1000]; initializeArray(); startThreads(); } public synchronized int getNextIndex() if(index < 1000) return(index++); else return(-1);

7.3 Java Example: public synchronized void addPartialSum(int partial_sum) { sum = sum + partial_sum; if(++threads_quit == number_of_threads) System.out.println("The sum of the numbers is " + sum); } private void initializeArray() int i; for(i = 0;i < 1000;i++) array[i] = i; public void startThreads() int i = 0; for(i = 0;i < 10;i++) AdderThread at = new AdderThread(this,i); at.start(); Adder a = new Adder();

7.3 Java Example: public static void main(String args[])
class AdderThread extends Thread { int partial_sum = 0; Adder parent; int number; public AdderThread(Adder parent,int number) this.parent = parent; this.number = number; } public void run() int index = 0; while(index != -1) { partial_sum = partial_sum + parent.array[index]; index = parent.getNextIndex(); System.out.println("Partial sum from thread " + number + " is " + partial_sum); parent.addPartialSum(partial_sum);

Process creation The concept of a thread and its creation Pthread routines How data can be created as shared data The concept of a condition variable Parallel programming language constructs Dependency analysis using Bernstein’s conditions Code examples

Further Reading There are a multitude of invented parallel programming languages Several are based upon FORTRAN and C++ Pthreads and multithreaded programming is covered in several books. Java is also an area for further study. An excellent starting point to learn java is where you can find several tutorials

Distributed Shared Memory
Chapter 9

Introduction Distributed Shared Memory
Making the main memory of a cluster of computers look as though it is a single memory with a single address space. Then you can use shared memory programming techniques.

1. Distributed Shared Memory
Still need messages or mechanisms to get data to processor, but these are hidden from the programmer:

Advantages of DSM System scalable Hides the message passing - do not explicitly specific sending messages between processes Can use simple extensions to sequential programming Can handle complex and large data bases without replication or sending the data to processes

Disadvantages of DSM May incur a performance penalty Must provide for protection against simultaneous access to shared data (locks, etc.) Little programmer control over actual messages being generated Performance of irregular problems in particular may be difficult

2. Implementing Distributed Shared Memory
Hardware Special network interfaces and cache coherence circuits Software Modifying the OS kernel Adding a software layer between the operating system and the application - most convenient way for teaching purposes

2.1 Software DSM Systems Page based Shared variable approach
Using the system’s virtual memory Shared variable approach Using routines to access shared variables Object based Shared data within collection of objects. Access to shared data through object oriented discipline (ideally)

2.1 Software DSM Systems Software Page Based DSM Implementation

2.1 Software DSM Systems Some Software DSM Systems Treadmarks JIAJIA
Page based DSM system Apparently not now available JIAJIA C based Some users have said it required significant modifications to work (in message-passing calls) Adsmith object based C++ library routines

2.2 Hardware DSM Implementation
Special network interfaces and cache coherence circuits are added to the system to make a memory reference to a remote look like a reference to a local memory location Typical examples are in clusters of SMP machines

2.3 Managing Shared Data There are several ways that a processor could be given access to shared data The simplest solution is to have a central server responsible for all read and write routines. Problem – the server is the bottleneck Solution?

2.3 Managing Shared Data Multiple copies of data.
Allows simultaneous access by different processors. How do you maintain these copies using a coherence policy? One option is Multiple Reader/Single Writer When the writer changes the data they have 2 choices Update policy Invalidate policy

2.3 Managing Shared Data Another option is Multiple Reader / Multiple Writer This is the most complex scenario.

2.4 Multiple Reader/Single Writer Policy in a Page-Based System
Remember that a page holds more than one variable location. Problem… A and B can change different things in the same page How do you deal with the consistency?

3. Achieving Consistent Memory in a DSM System
The term memory consistency model addresses when the current value of a shared variable is seen by other processors. There are various models with decreasing constraints to provide higher performance.

Consistency Models Strict Consistency - Processors sees most recent update, i.e. read returns the most recent write to location. Sequential Consistency - Result of any execution same as an interleaving of individual programs. Relaxed Consistency- Delay making write visible to reduce messages.

Consistency Models (cont) Weak consistency - programmer must use synchronization operations to enforce sequential consistency when necessary. Release Consistency - programmer must use specific synchronization operators, acquire and release. Lazy Release Consistency - update only done at time of acquire.

Strict Consistency Every write immediately visible Disadvantages: number of messages, latency, maybe unnecessary.

Release Consistency An extension of weak consistency in which the synchronization operations have been specified: acquire operation - used before a shared variable or variables are to be read. release operation - used after the shared variable or variables have been altered (written) and allows another process to access to the variable(s) Typically acquire is done with a lock operation and release by an unlock operation (although not necessarily).

Release Consistency Arrows show messages

Lazy Release Consistency Messages only on acquire Advantages: Fewer Messages

4. Distributed Shared Memory Programming Primitives
4 fundamental primitives Process/thread creation (and termination) Shared-data creation Mutual-exclusion synchronization (controlled access to shared data) Process/thread and event synchronization.

4.1 Process Creation Simple routines such as:
dsm_spawn(filename, num_processes); dsm_wait();

4.2 Shared Data Creation Routines to construct shares data
dsm_malloc(); dsm_free();

4.3 Shared Data Access In a DSM system employing relaxed-consistency model (most DSM systems) dsm_lock(lock1) dsm_refresh(sum); *sum++; dsm_flush(sum); dsm_unlock(lock1);

4.4 Synchronization Access
Two types must be provided Global Synchronization Process-Pair Synchronization Typically done with identifiers to barriers dsm_barrier(identifier);

4.5 Features to Improve Performance
Overlapping computation with communication. dsm_prefetch(sum); This tells the system to get the variable because we will need it soon. Similar to speculative loads used in processor/cache algorithms.

4.5 Features to Improve Performance
Reducing the Number of Messages dsm_acquire(sum); *sum++; dsm_release(sum); Old Way: dsm_lock(lock1); dsm_refresh(sum); *sum++; dsm_flush(sum); dsm_unlock(lock1);

5. Distributed Shared Memory Programming
Uses the same concepts as shared memory programming. Message passing occurs that is hidden from the user, and thus there are additional inefficiency considerations. You may have to modify the code to request variables before you need them. Would be nice if the compiler could do this

6. Implementing a Simple DSM System
It is relatively straightforward to write your own simple DSM system. In this section we will review how this can be done.

6.1 User Interface Using Classes and Methods.
Using the OO methodology of C++ we might have a wrapper class SharedInteger sum = new SharedInteger(); This class could extend the Integer class to provide the following methods. sum.lock() sum.refresh() sum++ sum.flush() sum.unlock()

6.2 Basic Shared-Variable Implementation
Option 1: Centralized Server

Option 2: Multiple Servers

Option 3: Multiple Readers (1 owner)

Option 4: Migrating the owner

DSM Projects Write a DSM system in C++ using MPI for the underlying message-passing and process communication. (More advanced) One of the fundamental disadvantages of software DSM system is the lack of control over the underlying message passing. Provide parameters in a DSM routine to be able to control the message-passing. Write routines that allow communication and computation to be overlapped.

Parallel Computers Chapter 1.

Similar presentations

Presentation on theme: "Parallel Computers Chapter 1."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Parallel Computers Chapter 1.

Similar presentations

Presentation on theme: "Parallel Computers Chapter 1."— Presentation transcript:

Similar presentations

About project

Feedback