1. Solution of Sparse Linear Systems Numerical Simulation CSE245 Thanks to: Kin Sou, Deepak Ramaswamy, Michal Rewienski, Jacob White, Shihhsien Kuo and Karen Veroy and especially Luca Daniel

2. 2 Outline of today’s lecture Solution of Sparse Linear SystemsSolution of Sparse Linear Systems –Examples of Problems with Sparse Matrices Struts and joints, resistor grids, 3-D heat flowStruts and joints, resistor grids, 3-D heat flow –Tridiagonal Matrix Factorization –General Sparse Factorization Fill-in and ReorderingFill-in and Reordering Graph Based ApproachGraph Based Approach –Sparse Matrix Data Structures ScatteringScattering

3. 3 1 2 3 4 5 6 7 8 9 Sparse Matrices Applications Space Frame X X X X X X X X X X X X X X X X X X X X XX X X XX X Unknowns : Joint positions Equations : forces = 0 X Nodal Matrix X X X X X X X X X X X

4. 4 Unknowns : Node Voltages Equations : currents = 0 Sparse Matrices Applications Resistor Grid

5. Nodal Formulation Matrix non-zero locations for 100 x 10 Resistor Grid Sparse Matrices Applications Resistor Grid

6. Nodal Formulation Sparse Matrices Applications Temperature in a cube Temperature known on surface, determine interior temperature Circuit Model

7. 7 Outline Solution of Dense Linear SystemsSolution of Dense Linear Systems Solution of Sparse Linear SystemsSolution of Sparse Linear Systems –LU Factorization Reminder. –Example of Problems with Sparse Matrices Struts and joints, resistor grids, 3-d heat flowStruts and joints, resistor grids, 3-d heat flow –Tridiagonal Matrix Factorization –General Sparse Factorization Fill-in and ReorderingFill-in and Reordering Graph Based ApproachGraph Based Approach –Sparse Matrix Data Structures ScatteringScattering

8. Nodal Formulation Sparse Matrices Tridiagonal Example Matrix Form m How many operations to do LU factorization?

9. 9 Sparse Matrices GE Algorithm Tridiagonal Example For i = 1 to n-1 { “For each Row” For j = i+1 to n { “For each target Row below the source” For k = i+1 to n { “For each Row element beyond Pivot” } Pivot Multiplier Order n Operations!

10. 10 Outline Solution of Dense Linear SystemsSolution of Dense Linear Systems Solution of Sparse Linear SystemsSolution of Sparse Linear Systems –LU Factorization Reminder. –Example of Problems with Sparse Matrices Struts and joints, resistor grids, 3-d heat flowStruts and joints, resistor grids, 3-d heat flow –Tridiagonal Matrix Factorization –General Sparse Factorization Fill-in and ReorderingFill-in and Reordering Graph Based ApproachGraph Based Approach –Sparse Matrix Data Structures ScatteringScattering

11. 11 Sparse Matrices Sparse Matrix Fill-In Symmetric Diagonally Dominant Nodal Matrix Resistor Example Example 0

12. 12 Sparse Matrices Fill-In Example X X XX X= Non zero Matrix Non zero structureFill Ins XX X = fill in where 0 became non-zero

13. 13 Sparse Matrices Fill-In Second Example Fill-ins Propagate XX X X X XX X XX Fill-ins from Step 1 result in Fill-ins in step 2

14. 14 Sparse Matrices Fill-In Reordering 0 Fill-ins Node Reordering - Can reduce fill-in - Preserves properties (Symmetry, Diagonal Dominance) - Equivalent to swapping rows and columns No Fill-ins 0

15. 15 Sparse Matrices Fill-In Reordering Where can fill-in occur ? Multipliers Already Factored Possible Fill-in Locations Fill-in Estimate = (Non zeros in unfactored part of Row -i) (Non zeros in unfactored part of Col -i) Markowitz Markowitz product

16. 16 Sparse Matrices Fill-In Reordering Markowitz Reordering Greedy Algorithm !

17. 17 Sparse Matrices Fill-In Why only try diagonals ? Corresponds to node reordering in Nodal formulation 0 3 12 0 2 31 Reduces search cost Preserves Matrix Properties - Diagonal Dominance - Symmetry Reordering

18. 18 Sparse Matrices Fill-In Pattern of a Filled-in Matrix Very Sparse Dense

19. 19 Sparse Matrices Fill-In Unfactored Random Matrix Symmetric

20. 20 Sparse Matrices Fill-In Factored Random Matrix Symmetric

21. 21 Outline Solution of Dense Linear SystemsSolution of Dense Linear Systems Solution of Sparse Linear SystemsSolution of Sparse Linear Systems –LU Factorization Reminder. –Example of Problems with Sparse Matrices Struts and joints, resistor grids, 3-d heat flowStruts and joints, resistor grids, 3-d heat flow –Tridiagonal Matrix Factorization –General Sparse Factorization Fill-in and ReorderingFill-in and Reordering Graph Based ApproachGraph Based Approach –Sparse Matrix Data Structures ScatteringScattering

22. 22 X X X X X X XX X XX X X XX 1 2 4 3 5 One Node Per Matrix Row One Edge Per Off-diagonal Pair X X Sparse Matrices Matrix Graphs Construction Structurally Symmetric Matrices and Graphs

23. 23 Sparse Matrices Matrix Graphs Markowitz Products X X X X X X XX X XX X X XX X X 1 2 4 3 5 Markowitz Products =(Node Degree) 2

24. 24 X X X X X X XX X XX X X XX 1 2 4 3 5 Delete the node associated with pivot row “Tie together” the graph edges X X X X X Sparse Matrices Matrix Graphs Factorization X One Step of LU Factorization

25. 25 Sparse Matrices Matrix Graphs Example 12345 Graph Markowitz products ( = Node degree) 12345

26. 26 Sparse Matrices Matrix Graphs Example Swap 2 with 1 X X 2345 Graph 2 345 Eliminate 1 112

27. 27 Unknowns : Node Voltages Equations : currents = 0 Sparse Matrices Graphs Resistor Grid Example

28. 28 Sparse Matrices Matrix Graphs Grid Example

29. 29 Sparse Matrices For growth control? Or to avoid fill-ins? What should you pivot for? A) LU factorization applied to a strictly diagonally dominant matrix will never produce a zero pivot B) The matrix entries produced by LU factorization applied to a strictly diagonally dominant matrix will never increase by more than a factor 2 (n-1) [which is anyway the best you can do by pivoting for growth control] Bottom line: if your matrix is strictly diagonally dominant no need for numerical pivot for growth control!

30. Sparse Factorization Approach 1)Assume matrix requires NO numerical pivoting. Diagonally dominant or symmetric positive definite. 2)Pre-Process: Use Graphs to Determine Matrix OrderingUse Graphs to Determine Matrix Ordering Many graph manipulation tricks used.Many graph manipulation tricks used. Form Data Structure for Storing Filled-in MatrixForm Data Structure for Storing Filled-in Matrix Lots of additional non-zero addedLots of additional non-zero added 3)Put numerical values in Data Structure and factor Computation must be organized carefully!

31. Summary of Sparse Systems LU Factorization and Diagonal Dominance.LU Factorization and Diagonal Dominance. –Factor without numerical pivoting Sparse MatricesSparse Matrices –Struts, resistor grids, 3-d heat flow -> O(N) non-zeros Tridiagonal Matrix FactorizationTridiagonal Matrix Factorization –Factor in O(N) operations General Sparse FactorizationGeneral Sparse Factorization –Markowitz Reordering to minimize fill Graph Based ApproachGraph Based Approach –Factorization and Fill-in –Useful for estimating Sparse GE complexity

32. 32 Outline Solution of Sparse Linear SystemsSolution of Sparse Linear Systems –Example of Problems with Sparse Matrices –Tridiagonal Matrix Factorization –General Sparse Factorization Fill-in and ReorderingFill-in and Reordering Graph Based ApproachGraph Based Approach Summary of the AlgorithmSummary of the Algorithm –Sparse Matrix Data Structures Scattering and symbolic fill inScattering and symbolic fill in

33. 33 Sparse Matrices Sparse Data Structure Val 11 Col 11 Val 12 Col 12 Val 1K Col 1K Val 21 Col 21 Val 22 Col 22 Val 2L Col 2L Val N1 Col N1 Val N2 Col N2 Val Nj Col Nj 1 N Arrays of Data in a Row Vector of row pointers Matrix entries Column index Row pointers Column Indices Goal: Never store a 0 Never multiply by 0

34. 34 Sparse Matrices Sparse Data Structure Problem of Misses Eliminating Source Row i from Target row j Row i Row j Must read all the row j entries to find the 3 that match row i

35. 35 Sparse Matrices Sparse Data Structure Data on Misses Every Miss is an unneeded Memory Reference! More misses than ops!

36. 36 Sparse Matrices Sparse Data Structure Scattering for Miss Avoidance Row j Use target row approach – Row j is the target. 1) Read all the elements in Row j, and scatter them in an n-length vector 2) Access only the needed elements using array indexing!

37. 37 Sparse Matrices – Another Data Structure Orthogonal linked list But if fill in occurs, pointer structures change. What do we do? Pre-compute pivoting order and sparse fill in structure symbolically

38. 38 Summary of Sparse Systems LU Factorization and Diagonal Dominance.LU Factorization and Diagonal Dominance. –Factor without numerical pivoting –hard problems (ill-conditioned) Sparse MatricesSparse Matrices –Struts, resistor grids, 3-d heat flow -> O(N) nonzeros Tridiagonal Matrix FactorizationTridiagonal Matrix Factorization –Factor in O(N) operations General Sparse FactorizationGeneral Sparse Factorization –Markowitz Reordering to minimize fill Graph Based ApproachGraph Based Approach –Factorization and Fill-in –Useful for estimating Sparse GE complexity Sparse Data StructuresSparse Data Structures –Scattering and symbolic fill in