1. Embarrassingly Parallel (or pleasantly parallel)
Definition: Problems that scale well to thousands of processors
Characteristics
Domain divisible into a large number of independent parts.
Little or no communication between processors
Processor performs the same calculation independently
“Nearly embarrassingly parallel”
Communication is limited to distributing and gathering data
Computation dominates the communication

2. Embarrassingly Parallel Examples
Embarrassingly Parallel Application P1 P2 P3 P0
Send Data
Receive Data
Nearly Embarrassingly Parallel Application

3. Low Level Image Processing
Note: Does not include communication to a graphics adapter
Storage
A two dimensional array of pixels.
One bit, one byte, or three bytes may represent pixels
Operations may only involve local data
Image Applications
Shift: newX=x+delta; newY=y+delta
Scale: newX = x*scale; newY = y*scale
Rotate a point about the origin newX = x cosF+y sinF; newY=-xsinF+ycosF
Clip newX = x if minx<=x< maxx; 0 otherwise newY = y if miny<=y<=maxy; 0 otherwise

4. Non-trivial Image Processing
Smoothing
A function that captures important patterns, while eliminating noise or artifacts
Linear smoothing: Apply a linear transformation to a picture
Convolution: Pnew(x,y) = ∑j=0,m-1∑k=0,n-1 P(x,y,j,k)old f(j,k)
Edge Detection
A function that searches for discontinuities or variations in depth, surface, or color
Purpose: Significantly reduce follow-up processing
Uses: Pattern recognition and computer vision
One approach: differentiate to identify large changes
Pattern Matching
Match an image against a template or a group of features
Example: ∑i=0,X∑j=1,Y (Picture(x+I,y+i) – Template(x,y))
Note: This is another digital signal processing application

5. Array Storage
Row –major (left most dimensions) are stored one after another
Column-major (right most dimensions) are stored one after another
The C language stores arrays in row-major order, Matlab and Fortran use column-major order
Loops can be extremely slow in C if the outer loop processes columns due to the system memory cache operation
int A[2][3] = { {1, 2, 3}, {4, 5, 6} }; In memory:
int A[2][3][2] = {{{1,2}, {3,4}, {5,6}}, {{7,8}, {9,10}, {11,12}}};
In memory:
Translate multi-dimension indices to single dimension offsets
Two Dimensions: offset = row*COLS + column
Three Dimensions: offset = i*DIM2*DIM3 + j*DIM3+ k
What is the formula for four dimensions?

6. Process Partitioning 1024 Note: 128 rows per displayed cell Pixel 2053
Row 2, column 5
Rows 0: 2:
Pixel 21
Rows
0: 0-7
2: 8-15
3:16-23
768
Note: 128 columns per displayed cell
Partitioning might assign groups of rows or columns to processors

7. Typical Static Partitioning
Master
Scatter or broadcast the image along with assigned processor rows
Gather the updated data back and perform final updates if necessary
Slave
Receive Data
Compute translated coordinates
Perform collective gather operation
Questions
How does the master decide how much to assign to each processor?
Is the load balanced (all processors working equally)?
Notes on the Text shift example
Employs individual sends/receives, which is much slower
However, if coordinate positions change or results do not represent contiguous pixel positions, this might be required

8. Mandelbrot Set Implementation Complex plane Display
Definition: Those points C = (x,y) = x + iy in the complex plane that iterate with a function (normally: zn+1 = zn2 + C) converge to a finite value
Implementation
Z0 = * i
For each (x,y) from [-2,+2]
Iterate zn until either
The iteration stops when the iteration count reaches a limit (not in the set)
Zn is out of bounds ( |zn|>2 (in the set)
Save the iteration count which will map to a display color
Complex plane Display
horizontal axis: real values
vertical axis: imaginary values

9. Scaling and Zooming Display range of points Display range of pixels
From cmin = xmin + iymin to cmax = xmax + iymax
Display range of pixels
From pixel at (0,0) to the pixel at (ROWS,COLUMNS)
Pseudo code
For row = 0 to ROWS
For col = 0 to COLUMNS
cy = ymin+(ymax-ymin)* row/ROWS
cx = xmin+(xmax-xmin)* col/COLUMNS
color = mandelbrot(cx, cy)
picture[COLUMNS*row + col] = color

10. Pseudo code (mandelbrot(cx,cy))
SET z = zreal + i*zimaginary = 0 + i0
SET iterations = 0
DO SET z = z2 + C
// temp = zreal; zreal=zreal2–zimaginary2 + cx
// zimaginary = 2 * temp * zimaginary + cy
SET value = zreal2 + zimaginary2
iterations++
WHILE value<=4 and iterations < MAX
RETURN iterations
Notes:
The final iteration count determines each point’s color
Some points converge quickly; others slowly, and others not at all
Non-converging points are in the Mandelbrot Set (black on the previous slide)
Note 4½ = 2, so we don't need to compute the square root when setting value

11. Parallel Implementation
Both the Static and Dynamic algorithms are examples of load balancing
Load-balancing
Algorithms used to avoid processors from becoming idle
Note: A balanced load does NOT require even same work loads
Static Approach
The load is assigned once at the start of the run
Mandelbrot: assign each processor a group of rows
Deficiencies: Not load balanced
Dynamic Approach
The load is dynamically assigned during the run
Mandelbrot: Slaves ask for work when they complete a section

12. The Dynamic Approach The Master's work is increased somewhat
Must send rows when receive requests from slaves
Must be responsive to slave requests. A separate thread might help or the master can make use of MPI's asynchronous receive calls.
Termination
Slaves terminate when receiving "no work" indication in received messages
The master must not terminate until all of the slaves complete
Partitioning of the load
Master receives blocks of pixels, Slaves receive ranges of (x,y) ranges
Partitions can be in columns or in rows. Which is better?
Refinement: Ask for work before completion (double buffering)

13. Pseudo-code (Throw darts to converge at a solution)
Monte Carlo Methods
Pseudo-code (Throw darts to converge at a solution)
Compute a definite integral While more iterations needed pick a random point total += f(x) result = 1/iterations * total
Calculation of PI While more iterations needed Randomly pick a point If point is in circle within++ // (x2+y2<=1) Compute PI = 4 * within / iterations Using the upper right quadrant eliminates the 4 in the equation
Note: Parallel programs shouldn't use the standard random number generator
1/N ∑1N f(pick.x) (xmax – xmin)

14. Computation of PI ∫(1-x2)1/2dx = π; -1<=x<=1
Within if (point.x2 + point.y2) ≤ 1
Total points/Points within = Total Area/Area in shape
Questions:
How to handle the boundary condition?
What is the best accuracy that we can achieve?

15. Parallel Random Number Generator
Numbers of a pseudo random sequence are
Uniformly, large period, repeatable, statistically independent
Each processor must generate a unique sequence
Accuracy depends upon random sequence precision
Sequential linear generator (a and m are prime; c=0)
Xi+1 = (axi +c) mod m (ex: a=16807, m=231 – 1, c=0)
Many other generators are possible
Parallel linear generator with unique sequences
Xi+k = (Axi + C) mod m where k is the "jump" constant
A=ak, C=c (ak-1 + ak-2 + … + a1 + a0)
if k = P, we can compute A and C and the first k random numbers to get started x1 x2
xP-1 xP
xP+1
x2P-2
x2P-1
Parallel Random Sequence