1. Embarrassingly Parallel Computations
Chapter 3

2. 1. Ideal Parallel Computation
In this chapter, we will consider applications where there is minimal interaction between the slave processes.
Slaves may be statically assigned or dynamically assigned.
Load balancing techniques can offer improved performance.
We will introduce simple load balancing here
Chapter 7 will cover load balancing when the slaves must interact.

3. 1. Ideal Parallel Computation
Parallelizing these should be obvious and requires no special techniques or algorithms.
A truly embarrassingly parallel computation requires no communication. (SPMD Model)

4. 1. Ideal Parallel Computation
Nearly embarrassingly parallel computations
Require results to be distributed and collected in some way.
A common approach is the Master-Slave organization.

5. 2. Embarrassingly Parallel Examples
2.1 Geometrical Transformation of Images
The most basic way to store a computer image is a pixmap (each pixel is stored in a 2D array)
Geometrical transformations require mathematical operations to be performed on the coordinates of each pixel without affecting its value.

6. 2.1 Geometrical Transformation of Images
There are several different transformations
Shifting
Object is shifted Dx in the x dimension and Dy in the y dimension:
x’=x+ Dx
y’=y+ Dy
Where x and y are the original and x’ and y’ are the new coordinates.
Scaling
Object scaled by factor Sx in the x-direction and Sy in the y-direction
x’=xSx
y’=ySy

7. 2.1 Geometrical Transformation of Images
Rotation
Object rotated through an angle q about the origin of the coordinate system:
x’=x cosq + y sinq
y’=-x sinq + y cosq
Clipping
Deletes from the displayed picture those points outside the defined area.

8. 2.1 Geometrical Transformation of Images
The input data is the pixmap (bitmap)
Typically held in a file, then copied into an array.
The master is concerned with dividing up the array among the processors.
There are two methods:
Blocks
Stripes

9. 2.1 Geometrical Transformation of Images

10. 2.1 Geometrical Transformation of Images

11. 2.1 Geometrical Transformation of Images: Master – pseudo code
for (i = 0, row = 0; i < 48; i++, row = row + 10)
/* for each process*/
send(row, Pi); /* send row no.*/
for (i = 0; i < 480; i++)
/* initialize temp */
for (j = 0; j < 640; j++)
temp_map[i][j] = 0;
for (i = 0; i < (640 * 480); i++) {
/* for each pixel */
recv(oldrow,oldcol,newrow,newcol, PANY);
/* accept new coords */
if !((newrow < 0)||(newrow >= 480)||(newcol < 0)||(newcol >= 640))
temp_map[newrow][newcol]=map[oldrow][oldcol]; }
for (i = 0; i < 480; i++) /* update bitmap */
map[i][j] = temp_map[i][j];

12. 2.1 Geometrical Transformation of Images: Slave – pseudo code
recv(row, Pmaster); /* receive row no. */
for (oldrow = row; oldrow < (row + 10); oldrow++)
for (oldcol = 0; oldcol < 640; oldcol++) {/* transform coords */
newrow = oldrow + delta_x; /* shift in x direction */
newcol = oldcol + delta_y; /* shift in y direction */
send(oldrow,oldcol,newrow,newcol, Pmaster);/* coords to master */ }

13. 2.1 Geometrical Transformation of Images: Analysis
Sequential: If each pixel required 1 time step then
ts = n2
So, time complexity is O(n2)
Recall that for parallel
tp = tcomp + tcomm
tcomm = tstartup + mtdata = O(m)
tcomp = 2(n2/p) = O(n2/p)
tp = O(n2)
What’s the problem?

14. 2.1 Geometrical Transformation of Images: Analysis
The constants for the communication (for parallel) far exceed the constants in the computation (for sequential)
Makes it relatively impractical
This problem is probably better suited for a shared memory system.

15. 2.2 Mandelbrot Set
Def: set of points in the plane that are quasi-stable when computed by iterating the function:
zk+1 = zk2+c
Note:
z is complex, and c is complex.
The initial value for z is 0
Iterations continue until
the magnitude of z is greater than 2
or the number of iterations reaches some limit

16. 2.2 Mandelbrot Set

17. 2.2 Mandelbrot Set: Sequential
int cal_pixel(complex c) {
int count, max;
complex z;
float temp, lengthsq;
max = 256;
z.real = 0; z.imag = 0;
count = 0; /* number of iterations */
do {
temp = z.real * z.real - z.imag * z.imag + c.real;
z.imag = 2 * z.real * z.imag + c.imag;
z.real = temp;
lengthsq = z.real * z.real + z.imag * z.imag;
count++;
} while ((lengthsq < 4.0) && (count < max));
return count; }

18. 2.2 Mandelbrot Set: Parallel
Static Task Assignment – Master
for (i = 0, row = 0; i < 48; i++, row = row + 10)
/* for each process*/
send(&row, Pi); /* send row no.*/
for (i = 0; i < (480 * 640); i++) {
/* from processes, any order */
recv(&c, &color, PANY); /*receive coords & colors */
display(c, color); /* display pixel on screen */ }

19. 2.2 Mandelbrot Set: Slave recv(&row, Pmaster); /* receive row no. */
for (x = 0; x < disp_width; x++)
/* screen coordinates x and y */
for (y = row; y < (row + 10); y++) {
c.real = min_real + ((float) x * scale_real);
c.imag = min_imag + ((float) y * scale_imag);
color = cal_pixel(c);
send(&c, &color, Pmaster);/* send coords, color to master */ }

20. 2.2 Mandelbrot Set
Dynamic Task Assignment
Work Pool/Processor Farms

21. Master count = 0; /* counter for termination*/
row = 0; /* row being sent */
for (k = 0; k < procno; k++) {
/* assuming procno<disp_height */
send(&row, Pk, data_tag); /* send initial row to process */
count++; /* count rows sent */
row++; /* next row */ }
do {
recv (&slave, &r, color, PANY, result_tag);
count--; /* reduce count as rows received */
if (row < disp_height) {
send (&row, Pslave, data_tag); /* send next row */
count++;
} else
send (&row, Pslave, terminator_tag); /* terminate */
rows_recv++;
display (r, color); /* display row */
} while (count > 0);

22. 2.2 Mandelbrot Set: Slave recv(y, Pmaster, ANYTAG, source_tag);
/* receive 1st row to compute */
while (source_tag == data_tag) {
c.imag = imag_min + ((float) y * scale_imag);
for (x = 0; x < disp_width; x++) { /* compute row colors */
c.real = real_min + ((float) x * scale_real);
color[x] = cal_pixel(c); }
send(&i, &y, color, Pmaster, result_tag);
/* row colors to master */
recv(y, Pmaster, source_tag);
/* receive next row */ };

23. 2.2 Mandelbrot Set

24. 2.2 Mandelbrot Set: Analysis
Sequential
ts= max x n = O(n)
Parallel
Phase 1: Communication Out
tcomm=s(tstartup+tdata)
It could be possible to use a scatter routine here which would cut the number of startup times.
Phase 2: Computation
(max x n)/s
Phase 3: Communication Back in
tcomm2= (n/s)(tstartup+tdata)
Overall
tp <= (max x n)/s + (n/s + s)(tstartup+tdata)

25. 2.3 Monte Carlo Methods
The basis of Monte Carlo methods is the use of random selections in calculations.
Example – Calculate p
A circle is formed within a square
The circle has unit radius
Therefore the square has sides of length 2
Area of the square is 4

26. 2.3 Monte Carlo Methods

27. 2.3 Monte Carlo Methods
The ratio of the area of the circle to the square is
(p r2)/2x2 = p/4
Points within the square are chosen randomly and a score is kept of how many points happen to lie within the circle
The fraction of points within the circle will be p/4, given a sufficient number of randomly selected samples.

28. 2.3 Monte Carlo Methods Random Number Generation
The most popular way of creating a pueudorandom number sequence:
x1, x2, x3, …, xi-1, xi, xi+1, …, xn-1, xn,
is by evaluating xi+1 from a carefully chosen function of xi, often of the form
xi+1 = (axi + c) mod m
where a, c, and m are constants chosen to create a sequence that has similar properties to truly random sequences.

29. 2.3 Monte Carlo Methods Parallel Random Number Generation
It turns out that
xi+1 = (axi + c) mod m
xi+k = (Axi + C) mod m
where
A = ak mod m,
C = c(ak-1 + an-2 + … + a1 + a0) mod m,
and k is a selected “jump” constant.

30. 5. Summary This Chapter introduced the following concepts:
An ideal embarrassingly parallel computation
Embarrassingly parallel problems and analyses
Partitioning a two-dimensional data set
Work pool approach to achieve load balancing
Counter termination algorithm
Monte Carlo methods
Parallel random number generation