1. CS 584 l Assignment

2. Systems of Linear Equations l A linear equation in n variables has the form l A set of linear equations is called a system. l A solution exists for a system iff the solution satisfies all equations in the system. l Many scientific and engineering problems take this form. a 0 x 0 + a 1 x 1 + … + a n-1 x n-1 = b

3. Solving Systems of Equations l Many such systems are large. –Thousands of equations and unknowns a 0,0 x 0 + a 0,1 x 1 + … + a 0,n-1 x n-1 = b 0 a 1,0 x 0 + a 1,1 x 1 + … + a 1,n-1 x n-1 = b 1 a n-1,0 x 0 + a n-1,1 x 1 + … + a n-1,n-1 x n-1 = b n-1

4. Solving Systems of Equations l A linear system of equations can be represented in matrix form a 0,0 a 0,1 … a 0,n-1 x 0 b 0 a 1,0 a 1,1 … a 1,n-1 x 1 b 1 a n-1,0 a n-1,1 … a n-1,n-1 x n-1 b n-1 = Ax = b

5. Solving Systems of Equations l Solving a system of linear equations is done in two steps: –Reduce the system to upper-triangular –Use back-substitution to find solution l These steps are performed on the system in matrix form. –Gaussian Elimination, etc.

6. Solving Systems of Equations l Reduce the system to upper-triangular form l Use back-substitution a 0,0 a 0,1 … a 0,n-1 x 0 b 0 0 a 1,1 … a 1,n-1 x 1 b 1 0 0 … a n-1,n-1 x n-1 b n-1 =

7. Reducing the System l Gaussian elimination systematically eliminates variable x[k] from equations k+1 to n-1. –Reduces the coefficients to zero l This is done by subtracting a appropriate multiple of the k th equation from each of the equations k+1 to n-1

8. Procedure GaussianElimination(A, b, y) for k = 0 to n-1 /* Division Step */ for j = k + 1 to n - 1 A[k,j] = A[k,j] / A[k,k] y[k] = b[k] / A[k,k] A[k,k] = 1 /* Elimination Step */ for i = k + 1 to n - 1 for j = k + 1 to n - 1 A[i,j] = A[i,j] - A[i,k] * A[k,j] b[i] = b[i] - A[i,k] * y[k] A[i,k] = 0 endfor end

9. Parallelizing Gaussian Elim. l Use domain decomposition –Rowwise striping l Division step requires no communication l Elimination step requires a one-to-all broadcast for each equation. l No agglomeration l Initially map one to to each processor

13. Communication Analysis l Consider the algorithm step by step l Division step requires no communication l Elimination step requires one-to-all bcast –only bcast to other active processors –only bcast active elements l Final computation requires no communication.

14. Communication Analysis l One-to-all broadcast –log 2 q communications –q = n - k - 1 active processors l Message size –q active processors –q elements required T = (t s + t w q)log 2 q

15. Computation Analysis l Division step –q divisions l Elimination step –q multiplications and subtractions l Assuming equal time --> 3q operations

16. Computation Analysis l In each step, the active processor set is reduced by one resulting in: 2/)1(3 1 1 0      nnCompTime kn n k

17. Can we do better? l Previous version is synchronous and parallelism is reduced at each step. l Pipeline the algorithm l Run the resulting algorithm on a linear array of processors. l Communication is nearest-neighbor l Results in O(n) steps of O(n) operations

18. Pipelined Gaussian Elim. l Basic assumption: A processor does not need to wait until all processors have received a value to proceed. l Algorithm –If processor p has data for other processors, send the data to processor p+1 –If processor p can do some computation using the data it has, do it. –Otherwise, wait to receive data from processor p-1

21. Conclusion l Using a striped partitioning method, it is natural to pipeline the Gaussian elimination algorithm to achieve best performance. l Pipelined algorithms work best on a linear array of processors. –Or something that can be linearly mapped l Would it be better to block partition? –How would it affect the algorithm?