1. CS 484

2. Iterative Methods n Gaussian elimination is considered to be a direct method to solve a system. n An indirect method produces a sequence of values that converges to the solution of the system. n Computation is halted in an indirect algorithm when a specified accuracy is reached.

3. Why Iterative Methods? n Sometimes we don't need to be exact. –Input has inaccuracy, etc. –Only requires a few iterations n If the system is sparse, the matrix can be stored in a different format. n Iterative methods are usually stable. –Errors are dampened

4. Iterative Methods n Consider the following system. 7 -6 x 1 3 -8 9 x 2 -4 = n Now write out the equations –solve for the i th unknown in the i th equation x 1 = 6/7 x 2 + 3/7 x 2 = 8/9 x 1 - 4/9

5. Iterative Methods n Come up with an initial guess for each x i n Hopefully, the equations will produce better values and which can be used to calculate better values, etc. and will converge to the answer.

6. Iterative Methods n Jacobi iteration –Use all old values to compute new values k x1 x2 0 0.00000 0.00000 10 0.14865 -0.19820 20 0.18682 -0.24908 30 0.19662 -0.26215 40 0.19913 -0.26551 50 0.19977 -0.26637

7. Jacobi Iteration n The ith equation has the form n Which can be rewritten as: ][][],[ 1 0 ibjxjiA n j                ij jxjiAib iiA ix][],[][ ],[ 1 ][

8. Jacobi Iteration n The vector (b - Ax) is zero when and if we get the exact answer. n Define this vector to be the residual r n Now rewrite the solution equation ][ ],[ ][ ][ix iiA ir ix 

9. void Jacobi(float A[][], float b[], float x[], float epsilon) { int k = 0; float x1[]; float r[]; float r0norm; // Randomly select an initial x vector r = b - Ax; // This involves matrix-vector mult etc. r0norm = ||r|| 2 ; // This is basically the magnitude while (||r|| 2 > epsilon * r0norm) { for (j = 0; j < n; j++) x1[j] = r[j] / A[j,j] + x[j]; r = b - Ax; } x = x1; }

10. Parallelization of Jacobi n 3 main computations per iteration n Inner product (2 norm) n Loop calculating x[j]s n Matrix-vector mult. to calculate r n If A[j,j], b[j], & r[j] are on the same proc. n Loop requires no communication n Inner product and Matrix-vector mult require communication.

11. Inner Product (2 norm of residual) n Suppose data is distributed row-wise n Inner product (2 norm) is simply dot product –IP = Sum(x[j] * x[j]) n This only requires a global sum collapse –O(log p) Axb= P0 P1 P2 P3

12. Matrix-Vector Multiplication n Again data is distributed row-wise n Each proc. requires all of the elements in the vector to calculate their part of the resulting answer. n This results in all to all gather –O(p log p)

13. Jacobi Iteration n Resulting cost for float (4 bytes) –T comm = #iterations * (T IP + T MVM ) –T IP = log p * (t s + t w * 4) –T MVM = p log p * (t s + t w * nrows/p * 4)

14. Iterative Methods n Gauss-Seidel –Use the new values as soon as available k x1 x2 0 0.00000 0.00000 10 0.21977 -0.24909 20 0.20130 -0.26531 30 0.20008 -0.26659 40 0.20000 -0.26666

15. Gauss-Seidel Iteration n The basic Gauss-Seidel iteration is                1 0 1 1 1 ],[][],[][][ ],[ 1 ][ i j n ij kkk jiAjxjiAjxib iiA ix

16. Gauss-Seidel Iteration n Rule: Always use the newest values of the previously computed variables. n Problem: Sequential? n Gauss-Seidel is indeed sequential if the matrix is dense. n Parallelism is a function of the sparsity and ordering of the equation matrix.

17. Gauss-Seidel Iteration n We can increase possible parallelism by changing the numbering of a system.

18. Parallelizing Red-Black GSI n Partitioning? n Communication? Block checkerboard. 2 phases per iteration. 1- compute red cells using values from black cells 2- compute black cells using values from red cells Communication is required for each phase.

19. Partitioning P0P0 P1P1 P2P2 P3P3

20. Communication P0P0 P1P1 P2P2 P3P3

21. Procedure Gauss-SeidelRedBlack while ( error > limit ) send black values to neighbors recv black values from neighbors compute red values send red values to neighbors recv red values from neighbors compute black values compute error /* only do every so often */ endwhile

22. Extending Red-Black Coloring n Goal: Produce a graph coloring scheme such that no node has a neighbor of the same color. n Simple finite element and finite difference methods produce graphs with only 4 neighbors. –Two colors suffice n What about more complex graphs?

23. More complex graphs n Use graph coloring heuristics. n Number the nodes one color at a time.

24. Successive Overrelaxation n Devised to speed up the convergence of Gauss-Seidel –apply extrapolation using a weighted average between current & previous iterates n Choose weighting that will accelerate the rate of convergence

25. SOR n Gauss-Seidel iteration n Don’t compute directly into x n Compute weighted average                1 0 1 1 1 ],[][],[][][ ],[ 1 ][ i j n ij kkk jiAjxjiAjxib iiA ix                1 0 1 1 1 ],[][],[][][ ],[ 1 ][ i j n ij kk jiAjxjiAjxib iiA i∂

26. SOR n SOR Equation Choose  in the range [0, 2] Choose  in the range [0, 2] –technically  < 1 is underrelaxation –if you choose  > 2 it won’t converge –if you choose  incorrectly it won’t converge

27. SOR Algorithm Choose an initial guess for x[i] for k = 1,2….. for i = 1,2, … n ∂ = 0 for j = 1,2,…., i-1 ∂ = ∂ + a[i,j] * x k [j] end for j = i+1, …, n ∂ = ∂ + a[i,j] * x k-1 [j] end ∂ = (b[i] - ∂) / a[i,i] x k [i] = x k-1 [i] + w * (∂ - x k-1 [i]) end check for convergence end Choose an initial guess for x[i] for k = 1,2….. for i = 1,2, … n ∂ = 0 for j = 1,2,…., i-1 ∂ = ∂ + a[i,j] * x k [j] end for j = i+1, …, n ∂ = ∂ + a[i,j] * x k-1 [j] end ∂ = (b[i] - ∂) / a[i,i] x k [i] = x k-1 [i] + w * (∂ - x k-1 [i]) end check for convergence end

28. Parallelizing SOR n Just like Gauss-Seidel n Create the dependency graph –Color –Number by color n Phases –Communicate nodes of previous phase –Compute nodes of this phase –Move to next phase

29. Conclusion n Iterative methods are used when an exact answer is not computable or needed. n Gauss-Seidel converges faster than Jacobi but parallelism is trickier. n Finite element codes are simply systems of equations –Solve with either Jacobi or Gauss-Seidel

30. Consider n Are systems of equations and finite element methods related?