Direct Methods for Sparse Linear Systems Lecture 4 Alessandra Nardi Thanks to Prof. Jacob White, Suvranu De, Deepak Ramaswamy, Michal Rewienski, and Karen Veroy

Last lecture review Solution of system of linear equations Existence and uniqueness review Gaussian elimination basics –GE basics –LU factorization –Pivoting

Outline Error Mechanisms Sparse Matrices –Why are they nice –How do we store them –How can we exploit and preserve sparsity

Error Mechanisms Round-off error –Pivoting helps Ill conditioning (almost singular) –Bad luck: property of the matrix –Pivoting does not help Numerical Stability of Method

Ill-Conditioning : Norms Norms useful to discuss error in numerical problems Norm

L 2 (Euclidean) norm : L 1 norm : L  norm : Unit circle Unit square 1 1 Ill-Conditioning : Vector Norms

Vector induced norm : Induced norm of A is the maximum “magnification” of by = max abs column sum = max abs row sum = (largest eigenvalue of A T A) 1/2 Ill-Conditioning : Matrix Norms

More properties on the matrix norm: Condition Number: -It can be shown that: -Large  (A) means matrix is almost singular (ill-conditioned)

What happens if we perturb b? Ill-Conditioning: Perturbation Analysis   (M) large is bad

What happens if we perturb M? Ill-Conditioning: Perturbation Analysis   (M) large is bad Bottom line: If matrix is ill-conditioned, round-off puts you in troubles

Geometric Approach is more intuitive When vectors are nearly aligned, Hard to decide how much of versus how much of Vectors are orthogonal Ill-Conditioning: Perturbation Analysis Vectors are nearly aligned

Numerical Stability Rounding errors may accumulate and propagate unstably in a bad algorithm. Can be proven that for Gaussian elimination the accumulated error is bounded

Summary on Error Mechanisms for GE Rounding: due to machine finite precision we have an error in the solution even if the algorithm is perfect –Pivoting helps to reduce it Matrix conditioning –If matrix is “good”, then complete pivoting solves any round-off problem –If matrix is “bad” (almost singular), then there is nothing to do Numerical stability –How rounding errors accumulate –GE is stable

LU – Computational Complexity Computational Complexity –O(n 3 ), where M: n x n We cannot afford this complexity Exploit natural sparsity that occurs in circuits equations –Sparsity: many zero elements –Matrix is sparse when it is advantageous to exploit its sparsity Exploiting sparsity: O(n 1.1 ) to O(n 1.5 )

LU – Goals of exploiting sparsity (1)Avoid storing zero entries –Memory usage reduction –Decomposition is faster since you do need to access them (but more complicated data structure) (2) Avoid trivial operations –Multiplication by zero –Addition with zero (3) Avoid losing sparsity

m Sparse Matrices – Resistor Line Tridiagonal Case

For i = 1 to n-1 { “ For each Row ” For j = i+1 to n { “For each target Row below the source” For k = i+1 to n { “For each Row element beyond Pivot” } Pivot Multiplier Order N Operations! GE Algorithm – Tridiagonal Example

Symmetric Diagonally Dominant Nodal Matrix 0 Sparse Matrices – Fill-in – Example 1

X X XX X= Non zero Matrix Non zero structureMatrix after one LU step XX Sparse Matrices – Fill-in – Example 1

Fill-ins Propagate XX X X X XX X XX Fill-ins from Step 1 result in Fill-ins in step 2 Sparse Matrices – Fill-in – Example 2

Node Reordering Can Reduce Fill-in - Preserves Properties (Symmetry, Diagonal Dominance) - Equivalent to swapping rows and columns 0 Fill-ins 0 No Fill-ins Sparse Matrices – Fill-in & Reordering

Exploiting and maintaining sparsity Criteria for exploiting sparsity: –Minimum number of ops –Minimum number of fill-ins Pivoting to maintain sparsity: NP-complete problem  heuristics are used –Markowitz, Berry, Hsieh and Ghausi, Nakhla and Singhal and Vlach –Choice: Markowitz 5% more fill-ins but Faster Pivoting for accuracy may conflict with pivoting for sparsity

Where can fill-in occur ? Multipliers Already Factored Possible Fill-in Locations Fill-in Estimate = (Non zeros in unfactored part of Row -1) (Non zeros in unfactored part of Col -1) Markowitz product Sparse Matrices – Fill-in & Reordering

Markowitz Reordering (Diagonal Pivoting) Greedy Algorithm (but close to optimal) ! Sparse Matrices – Fill-in & Reordering

Why only try diagonals ? Corresponds to node reordering in Nodal formulation Reduces search cost Preserves Matrix Properties - Diagonal Dominance - Symmetry Sparse Matrices – Fill-in & Reordering

Pattern of a Filled-in Matrix Very Sparse Dense Sparse Matrices – Fill-in & Reordering

Unfactored random Matrix

Sparse Matrices – Fill-in & Reordering Factored random Matrix

Sparse Matrices – Data Structure Several ways of storing a sparse matrix in a compact form Trade-off –Storage amount –Cost of data accessing and update procedures Efficient data structure: linked list

Sparse Matrices – Data Structure 1 Orthogonal linked list

Val 11 Col 11 Val 12 Col 12 Val 1K Col 1K Val 21 Col 21 Val 22 Col 22 Val 2L Col 2L Val N1 Col N1 Val N2 Col N2 Val Nj Col Nj 1 N Arrays of Data in a Row Vector of row pointers Matrix entries Column index Sparse Matrices – Data Structure 2

Eliminating Source Row i from Target row j Row i Row j Must read all the row j entries to find the 3 that match row i Every Miss is an unneeded memory reference (expensive!!!) Could have more misses than ops! Sparse Matrices – Data Structure Problem of Misses

Row j 1) Read all the elements in Row j, and scatter them in an n-length vector 2) Access only the needed elements using array indexing! Sparse Matrices – Data Structure Scattering for Miss Avoidance

X X X X X X XX X XX X X XX One Node Per Matrix Row One Edge Per Off-diagonal Pair X X Structurally Symmetric Matrices and Graphs Sparse Matrices – Graph Approach

X X X X X X XX X XX X X XX X X Markowitz Products =(Node Degree) 2 Sparse Matrices – Graph Approach Markowitz Products

X X X X X X XX X XX X X XX Delete the node associated with pivot row “Tie together” the graph edges X X X X X One Step of LU Factorization Sparse Matrices – Graph Approach Factorization

12345 Graph Markowitz products ( = Node degree) Sparse Matrices – Graph Approach Example

1345 Graph Swap 2 with 1 Sparse Matrices – Graph Approach Example

Gaussian Elimination Error Mechanisms –Ill-conditioning –Numerical Stability Gaussian Elimination for Sparse Matrices –Improved computational cost: factor in O(N 1.5 ) operations (dense is O(N 3 ) ) –Example: Tridiagonal Matrix Factorization O(N) –Data structure –Markowitz Reordering to minimize fill-ins –Graph Based Approach Summary