Direct Methods for Sparse Linear Systems Lecture 4 Alessandra Nardi Thanks to Prof. Jacob White, Suvranu De, Deepak Ramaswamy, Michal Rewienski, and Karen Veroy

Last lecture review Solution of system of linear equations Existence and uniqueness review Gaussian elimination basics GE basics LU factorization Pivoting

Outline Error Mechanisms Sparse Matrices Why are they nice How do we store them How can we exploit and preserve sparsity

Error Mechanisms Round-off error Ill conditioning (almost singular) Pivoting helps Ill conditioning (almost singular) Bad luck: property of the matrix Pivoting does not help Numerical Stability of Method

Ill-Conditioning : Norms Norms useful to discuss error in numerical problems Norm

Ill-Conditioning : Vector Norms L2 (Euclidean) norm : Unit circle L1 norm : 1 1 As the algebra on the slide shows the relative changes in the solution x is bounded by an A-dependent factor times the relative changes in A. The factor was historically referred to as the condition number of A, but that definition has been abandoned as then the condition number is norm-dependent. Instead the condition number of A is the ratio of singular values of A. Singular values are outside the scope of this course, consider consulting Trefethen & Bau. L norm : Unit square

Ill-Conditioning : Matrix Norms Vector induced norm : Induced norm of A is the maximum “magnification” of by = max abs column sum As the algebra on the slide shows the relative changes in the solution x is bounded by an A-dependent factor times the relative changes in A. The factor was historically referred to as the condition number of A, but that definition has been abandoned as then the condition number is norm-dependent. Instead the condition number of A is the ratio of singular values of A. Singular values are outside the scope of this course, consider consulting Trefethen & Bau. = max abs row sum = (largest eigenvalue of ATA)1/2

Ill-Conditioning : Matrix Norms More properties on the matrix norm: Condition Number: It can be shown that: Large k(A) means matrix is almost singular (ill-conditioned)

Ill-Conditioning: Perturbation Analysis What happens if we perturb b? As the algebra on the slide shows the relative changes in the solution x is bounded by an A-dependent factor times the relative changes in A. The factor was historically referred to as the condition number of A, but that definition has been abandoned as then the condition number is norm-dependent. Instead the condition number of A is the ratio of singular values of A. Singular values are outside the scope of this course, consider consulting Trefethen & Bau.  k(M) large is bad

Ill-Conditioning: Perturbation Analysis What happens if we perturb M?  k(M) large is bad As the algebra on the slide shows the relative changes in the solution x is bounded by an A-dependent factor times the relative changes in A. The factor was historically referred to as the condition number of A, but that definition has been abandoned as then the condition number is norm-dependent. Instead the condition number of A is the ratio of singular values of A. Singular values are outside the scope of this course, consider consulting Trefethen & Bau. Bottom line: If matrix is ill-conditioned, round-off puts you in troubles

Ill-Conditioning: Perturbation Analysis Geometric Approach is more intuitive Vectors are nearly aligned Vectors are orthogonal When vectors are nearly aligned, Hard to decide how much of versus how much of

Numerical Stability Rounding errors may accumulate and propagate unstably in a bad algorithm. Can be proven that for Gaussian elimination the accumulated error is bounded

Summary on Error Mechanisms for GE Rounding: due to machine finite precision we have an error in the solution even if the algorithm is perfect Pivoting helps to reduce it Matrix conditioning If matrix is “good”, then complete pivoting solves any round-off problem If matrix is “bad” (almost singular), then there is nothing to do Numerical stability How rounding errors accumulate GE is stable

LU – Computational Complexity O(n3), where M: n x n We cannot afford this complexity Exploit natural sparsity that occurs in circuits equations Sparsity: many zero elements Matrix is sparse when it is advantageous to exploit its sparsity Exploiting sparsity: O(n1.1) to O(n1.5)

LU – Goals of exploiting sparsity Avoid storing zero entries Memory usage reduction Decomposition is faster since you do need to access them (but more complicated data structure) (2) Avoid trivial operations Multiplication by zero Addition with zero (3) Avoid losing sparsity

Sparse Matrices – Resistor Line Tridiagonal Case m

GE Algorithm – Tridiagonal Example For i = 1 to n-1 { “For each Row” For j = i+1 to n { “For each target Row below the source” For k = i+1 to n { “For each Row element beyond Pivot” } Pivot Multiplier Order N Operations!

Sparse Matrices – Fill-in – Example 1 Nodal Matrix Recalling from lecture 2, the entries in the nodal matrix can be derived by noting that a resistor, as contributes to four locations in the nodal matrix as shown below. It is also resisting to note that Gii is equal to the sum of the conductances (one over resistance) incident at node i. Symmetric Diagonally Dominant

Sparse Matrices – Fill-in – Example 1 Matrix Non zero structure Matrix after one LU step X X X X X X During a step of LU factorization a multiple of a source row will be subtracted from a row below it. Since these two rows will not necessarily have non zeros in the same columns, the result of the subtraction might be to introduce additional non zeros into the target row. As a simple example, consider LU factoring The result is Notice that the factored matrix has a non zero entry in the bottom right corner, where as the original matrix did not. This changing of a zero entry to a non zero entry is referred to as a fill-in. X= Non zero

Sparse Matrices – Fill-in – Example 2 Fill-ins Propagate X X X X X X X In the example, the 4 x 4 mesh begins with 7 zeros. During the LU factorization, 5 of the zeros become non zero. What is of additional concern is the problem of fill-ins. The first step of LU factorization where a multiple of the first row is subtracted from the second row, generates fill-ins in the third and fourth column of row two. When multiples of row 2 are subtracted from row 3 and row 4, the fill-ins generated in row 2 generate second-level fill-ins in rows 3 and 4. X X X Fill-ins from Step 1 result in Fill-ins in step 2

Sparse Matrices – Fill-in & Reordering Fill-ins No Fill-ins In the context of the nodal equation formulation, renumbering the nodes seems like a simple operation to reduce fill-in, as selecting the node numbers was arbitrary to begin with. Keep in mind, however, that such a renumbering of nodes in the nodal equation formulation corresponds to swapping both rows and columns in the matrix. Node Reordering Can Reduce Fill-in - Preserves Properties (Symmetry, Diagonal Dominance) - Equivalent to swapping rows and columns

Exploiting and maintaining sparsity Criteria for exploiting sparsity: Minimum number of ops Minimum number of fill-ins Pivoting to maintain sparsity: NP-complete problem  heuristics are used Markowitz, Berry, Hsieh and Ghausi, Nakhla and Singhal and Vlach Choice: Markowitz 5% more fill-ins but Faster Pivoting for accuracy may conflict with pivoting for sparsity

Sparse Matrices – Fill-in & Reordering Where can fill-in occur ? Already Factored Possible Fill-in Locations Multipliers Fill-in Estimate = (Non zeros in unfactored part of Row -1) (Non zeros in unfactored part of Col -1) Markowitz product

Sparse Matrices – Fill-in & Reordering Markowitz Reordering (Diagonal Pivoting) In order to understand the Markowitz reordering algorithm, it is helpful to consider the cost of the algorithm. The first step is to determine the diagonal with the minimum Markowitz product. The cost of this step is where K is the average number of non zeros per row. The second step of the algorithm is to swap rows and columns in the factorization. A good data structure will make the swap inexpensively. The third step is to factor the reordered matrix and insert the fill-ins. If the matrix is very sparse, this third step will also be inexpensive. Since one must then find the diagonal in the updated matrix with the minimum Markowitz product, the products must be computed at a cost of It is possible to improve the situation by noting that very few Markowitz products will change during a single step of the factorization. The mechanics of such an optimization are easiest to see by examining the graphs of a matrix. Greedy Algorithm (but close to optimal) !

Sparse Matrices – Fill-in & Reordering Why only try diagonals ? Corresponds to node reordering in Nodal formulation 1 2 3 1 3 2 Reduces search cost Preserves Matrix Properties - Diagonal Dominance - Symmetry

Sparse Matrices – Fill-in & Reordering Pattern of a Filled-in Matrix Very Sparse Very Sparse Dense

Sparse Matrices – Fill-in & Reordering Unfactored random Matrix

Sparse Matrices – Fill-in & Reordering Factored random Matrix

Sparse Matrices – Data Structure Several ways of storing a sparse matrix in a compact form Trade-off Storage amount Cost of data accessing and update procedures Efficient data structure: linked list

Sparse Matrices – Data Structure 1 In order to store a sparse matrix efficiently, one needs a data structure which can represent only the matrix non zeros. One simple approach is based on the observation that each row of a sparse matrix has at least one non zero entry. Then one constructs one pair of arrays for each row, where the array part corresponds to the matrix entry and the entry’s column. As an example, consider the matrix The data structure for this example is Note that there is no explicit storage for the zeros Orthogonal linked list

Sparse Matrices – Data Structure 2 Vector of row pointers Arrays of Data in a Row Matrix entries Val 11 Val 12 Val 1K 1 Column index Col 11 Col 12 Col 1K Val 21 Val 22 Val 2L Col 21 Col 22 Col 2L In order to store a sparse matrix efficiently, one needs a data structure which can represent only the matrix non zeros. One simple approach is based on the observation that each row of a sparse matrix has at least one non zero entry. Then one constructs one pair of arrays for each row, where the array part corresponds to the matrix entry and the entry’s column. As an example, consider the matrix The data structure for this example is Note that there is no explicit storage for the zeros Val N1 Val N2 Val Nj N Col N1 Col N2 Col Nj

Sparse Matrices – Data Structure Problem of Misses Eliminating Source Row i from Target row j Row i Row j In order to store a sparse matrix efficiently, one needs a data structure which can represent only the matrix non zeros. One simple approach is based on the observation that each row of a sparse matrix has at least one non zero entry. Then one constructs one pair of arrays for each row, where the array part corresponds to the matrix entry and the entry’s column. As an example, consider the matrix The data structure for this example is Note that there is no explicit storage for the zeros Must read all the row j entries to find the 3 that match row i Every Miss is an unneeded memory reference (expensive!!!) Could have more misses than ops!

Sparse Matrices – Data Structure Scattering for Miss Avoidance Row j In order to store a sparse matrix efficiently, one needs a data structure which can represent only the matrix non zeros. One simple approach is based on the observation that each row of a sparse matrix has at least one non zero entry. Then one constructs one pair of arrays for each row, where the array part corresponds to the matrix entry and the entry’s column. As an example, consider the matrix The data structure for this example is Note that there is no explicit storage for the zeros 1) Read all the elements in Row j, and scatter them in an n-length vector 2) Access only the needed elements using array indexing!

Sparse Matrices – Graph Approach Structurally Symmetric Matrices and Graphs 1 X X X X X X X 2 X X X X 3 X X X The graph has The graph has two important properties 1) The node degree squared yields the Markowitz product. 2) The graph can easily be updated after one step of factorization. The graph makes efficient a two-step approach to factoring a structurally symmetric matrix. First one determines an ordering which produces little fill by using the graph. Then, one numerically factors the matrix in the graph-determined order. 4 X X X 5 One Node Per Matrix Row One Edge Per Off-diagonal Pair

Sparse Matrices – Graph Approach Markowitz Products X 1 2 4 3 5 Markowitz Products = (Node Degree)2 That the ith node degree squared is equal to the Markowitz product associated with the ith diagonal is easy to see. The node degree is the number of edges emanating from the node, and each edge represents both an off-diagonal row entry and an off-diagonal column entry. Therefore, the number of off-diagonal row entries multiplied by the number of off-diagonal column entries is equal to the node degree squared.

Sparse Matrices – Graph Approach Factorization One Step of LU Factorization 1 X X X X X X X 2 X X X X X 3 X X X 4 X X X X X One step of LU factorization requires a number of floating point operations and produces a reduced matrix, as below After step i in the factorization, the unfactored portion of the matrix is smaller of size (i-1) x (i-1), and may be denser if there are fill-ins. The graph can be used to represent the location of non zeros in the unfactored portion of the matrix, but two things must change. 1) A node must be removed as the unfactored portion has one fewer row. 2) The edges associated with fill-ins must be added. In the animation, we show by example how the graph is updated during a step of LU factorization. We can state the manipulation precisely by noting that if row i is eliminated in the matrix, the node i must be eliminated from the graph. In addition, all nodes adjacent to node i ( adjacent nodes are ones connected by an edge) will be made adjacent to each other by adding the necessary edges. The added edges represent fill-in. 5 factored Delete the node associated with pivot row “Tie together” the graph edges factored

Sparse Matrices – Graph Approach Example 1 2 3 4 5 Graph Markowitz products ( = Node degree) 1 2 3 4 5

Sparse Matrices – Graph Approach Example Swap 2 with 1 1 3 4 5 Graph Examples that factor with no fill-in Tridiagonal Another ordering for the tridiagonal matrix that is more parallel 1 7 A B C D E F G 1 2 3 A B C D E F G 4 5 A C E G A E

Summary Gaussian Elimination Error Mechanisms Ill-conditioning Numerical Stability Gaussian Elimination for Sparse Matrices Improved computational cost: factor in O(N1.5) operations (dense is O(N3) ) Example: Tridiagonal Matrix Factorization O(N) Data structure Markowitz Reordering to minimize fill-ins Graph Based Approach