1. All-to-All Pattern
A pattern where all (slave) processes can communicate with each other
Somewhat the worst case scenario!
ITCS 4/5145 Parallel Computing, UNC-Charlotte, B. Wilkinson, slides3b.ppt Revised Oct 17, 2013

2. All-to-All communication
Some problems requires this.
Examples
N-body problem
Solving dense system of linear equations

3. Gravitational N-Body Problem
Finding positions and movements of bodies in space subject to gravitational forces from other bodies.
Use Newtonian laws of physics:
Equations: Gravitational force between two bodies of masses ma and mb is:
G, gravitational constant. r distance between bodies.
Subject to forces, body accelerates according to Newton’s 2nd law:
F = ma
M, mass of body, F force it experiences, a resultant acceleration.

4. Details Force – First compute the force:
Force – First compute the force:
Velocity -- Let time interval be Dt. From:
for a body of mass m, new velocity is:
where vt+1 is velocity at time t + 1 and vt is velocity at time t.
Position -- Over interval Dt, position changes by
where xt is its position at time t.
Once bodies move to new positions, forces change. Computation has to be repeated.

5. This then gives the velocity and positions in three directions.
This then gives the velocity and positions in three directions.

6. This then gives the velocity and positions in two directions.
two
This then gives the velocity and positions in two directions.

7. Assignment 4 specifies two-dimensional space -- a little easier to visualize.y x
Force on body r
Another body
Add the force cause by each body in x and x directions
Moves
Movement

8. Data for 2-D Gravitational N-body problem (Assignment 4)
Table used to hold initial and computed data over time steps:
Body
Mass
Position in x direction
Position in y direction
Velocity in x direction
Velocity in y direction 1 2 … N
On each iteration, position and velocities are updated.
Table can be used to display movement of bodies

9. Sequential Code. The overall gravitational N-body computation can be described by the following steps:
for (t = 0; t < tmax; t++) { //for each time period for (i = 0; i < N; i++) { //for body i, calculate force on body due to other bodies for (j = 0; j < N; j++) { if (i != j) { // for different bodies x_diff = ... ; // compute distance between body i and body j in x direction y_diff = ... ; // compute distance between body i and body j in y direction r = ... ; //compute distance r F = ... ; // compute force on bodies Fx[i] += ... ; // resolve and accumulate force in x direction Fy[i] += … ; // resolve and accumulate force in y direction } for (i = 0; i < N; i++) { // for each body, update positions and velocity A[i][x_velocity]= ... ; // new velocity in x direction A[i][y_velocity]= ... ; // new velocity in y direction A[i][x_position] = ... ; // new position in x direction A[i][y_position] = ... ; // new position in y direction } // end time period

10. Time complexity
Brute-force sequential algorithm is an O(N2) algorithm for one iteration as each of the N bodies is influenced by each of the other N - 1 bodies.
For t iterations, O(N2t)
Not feasible to use this direct algorithm for most interesting N-body problems where N is very large.

11. Reducing time complexity
Time complexity can be reduced approximating a cluster of distant bodies as a single distant body with mass sited at the center of mass of the cluster:

12. Barnes-Hut Algorithm
Start with whole space in which one cube contains the bodies (or particles).
• First, this cube is divided into eight subcubes.
• If a subcube contains no particles, subcube deleted from further consideration.
• If a subcube contains one body, subcube retained.
• If a subcube contains more than one body, it is recursively divided until every subcube contains one body.

13. Creates an octtree - a tree with up to eight edges from each vertex (node).
Leaves represent cells each containing one body.
After tree constructed, total mass and center of mass of subcube stored at each vertex (node).

14. Force on each body obtained by traversing tree starting at root, stopping at a node when the clustering approximation can be used, e.g. when r is greater than some distance D.
Constructing tree requires a time of O(NlogN), and so does computing all the forces, so that overall time complexity of method is O(NlogN).

15. Example for 2-dimensional space
At each vertex, store coordinates of center of mass and total mass of bodies in space below
(bodies)
One body

16. Computing force on each body -- traverse tree starting at root, stopping at a node when clustering approximation can be used, i.e. when r is greater than some set distance D.
For each body
Mass and coordinates of center of mass of bodies in sub space

17. Orthogonal Recursive Bisection
An alternative way of dividing space.
(For 2-dimensional area) First, a vertical line found that divides area into two areas each with equal number of bodies. For each area, a horizontal line found that divides it into two areas each with equal number of bodies. Repeated as required.

18. Iterative synchronous patterns
When a pattern is repeated until some termination condition occurs.
Synchronization at each iteration, to establish termination condition, often a global condition.
Note this is actually two patterns joined together sequentially if we call iteration a pattern.
Pattern
Check termination condition
Repeat
Stop
Note these pattern names are our names.

19. Iterative synchronous all-to-all pattern
N-body problem needs an “iterative synchronous all-to-all” pattern, where on each iteration all the processes exchange data with each other:
Iterative synchronous all-to-all pattern
Repeat
Stop
Check termination condition
6a.19

20. Solving General System of Linear Equations
Some problems of this type require a number of iterations to converge on the solution – example:
Solving General System of Linear Equations
by iteration
Suppose equations are of a general form with n equations and n unknowns:
where the unknowns are x0, x1, x2, … xn-1 (0 <= i < n).
6a.2020

21. By rearranging the ith equation:
This equation gives xi in terms of the other unknowns.
Can be used as an iteration formula for each of the unknowns to obtain better approximations. Process i computes xi
6a.2121

22. Suppose each process computes one unknown. Pi computes xi
Process Pi needs unknowns from all other processes on each iteration P0
Pn-1
(Excluding Pi)
Computes: Pi
Needs iterative synchronous all-to-all pattern
6a.22

23. Jacobi Iteration
Name given to a computation that uses the previous iteration value to compute the next values.*
All values of x are updated together.
Convergence: Can be proven that the Jacobi method will converge if diagonal values of a have an absolute value greater than sum of absolute values of other a’s on row, i.e. if
This condition is a sufficient but not a necessary condition.
* Other (non-parallel) methods use some of the present iteration values to compute the present values, see later.
6a.2323

24. Termination
Simple, common approach is compare values computed in one iteration to values obtained from the previous iteration.
Terminate computation when all values are within given tolerance; i.e., when
However, this does not guarantee the solution to that accuracy. Why?
6a.2424

25. Convergence Rate
6a.25

26. Seeds “CompleteSynchGraph” Pattern
All-to-all pattern that includes a synchronous iteration feature to pass results of one iteration to all the nodes before the next iteration. Instead of sharing a pool of tasks to execute, workers gets replicas of the initial data set. At each iteration, workers synchronize and update their replicas and proceed to new computations. Master node and framework will not get control of the data flow until all the iterations done.

27. More information on using Seeds CompleteSynchGraph Pattern
Seeds CompleteSynchGraph tutorial: “Seeds Framework – The CompleteSynchGraph Template Tutorial,” Jeremy Villalobos and Yawo K. Adibolo, June 18, at (to be moved) Gives details with code for the Jacobi iteration method of solving system of linear equations

28. Notes on solution in CompleteSynchGraph Template Tutorial
Equations in matrix-vector form: AX = B where:
Converted to:
Xk = CXk-1 + D
where Xk is solution vector at iteration k
Xk-1 the solution vector at iteration k-1
C a matrix derived from input matrix A
D a vector derived from input vector B.
Each slave assigned one or more equations to solve
6a.28

29. MPI implementation of all-to-all pattern
MPI_Allgather() routine
MPI_AllGather broadcasts and gather values in one composite construction:
int MPI_Allgather(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, MPI_Comm comm)
6a.29

30. When does the MPI_AllGather() return?
MPI-AllGather() has the same effect as n MPI_Gather()’s executed, for root = 0 to n-1.
MPI_Gather has the same effect as each process executing an MPI_Send() and n MPI_Recv()s
Question
When does the MPI_AllGather() return?
Answer
6a.30

31. Questions