1. Direct Methods for Linear Systems Lecture 3 Alessandra Nardi Thanks to Prof. Jacob White, Suvranu De, Deepak Ramaswamy, Michal Rewienski, and Karen Veroy

2. Last lecture review Formulation of circuit equations –Conservation laws (KCL, KVL) –Branch constitutive equations (BCE) –KCL, KVL, BCE combined in different ways: STA MNA

3. Outline Systems of linear equations –Existence and uniqueness review –Gaussian Elimination basics LU factorization Pivoting

4. Systems of linear equations Problem to solve: M x = b Given M x = b : –Is there a solution? –Is the solution unique?

5. Systems of linear equations Find a set of weights x so that the weighted sum of the columns of the matrix M is equal to the right hand side b

6. Systems of linear equations - Existence A solution exists when b is in the span of the columns of M A solution exists if: There exist weights, x 1, …., x N, such that:

7. Systems of linear equations - Uniqueness A solution is unique only if the columns of M are linearly independent. Then: Mx = b  Mx + My= b  M(x+y) = b Suppose there exist weights, y 1, …., y N, not all zero, such that:

8. Systems of linear equations Square matrices Given Mx = b, where M is square – If a solution exists for any b, then the solution for a specific b is unique. For a solution to exist for any b, the columns of M must span all N-length vectors. Since there are only N columns of the matrix M to span this space, these vectors must be linearly independent. A square matrix with linearly independent columns is said to be nonsingular.

9. Application Problems Matrix is n x n Often symmetric and diagonally dominant Nonsingular of real numbers

10. Methods for solving linear equations Direct methods: find the exact solution in a finite number of steps Iterative methods: produce a sequence a sequence of approximate solutions hopefully converging to the exact solution

11. Gaussian Elimination Basics Gaussian Elimination Method for Solving M x = b A “Direct” Method Finite Termination for exact result (ignoring roundoff) Produces accurate results for a broad range of matrices Computationally Expensive

12. Gaussian Elimination Basics Reminder by 3x3 example

13. Gaussian Elimination Basics – Key idea Use Eqn 1 to Eliminate x 1 from Eqn 2 and 3

14. GE Basics – Key idea in the matrix MULTIPLIERSPivot Remove x1 from eqn 2 and eqn 3

15. GE Basics – Key idea in the matrix Pivot Multiplier Remove x2 from eqn 3

16. GE Basics – Simplify the notation Remove x1 from eqn 2 and eqn 3

17. Pivot Multiplier GE Basics – Simplify the notation Remove x2 from eqn 3

18. GE Basics – GE yields triangular system Altered During GE ~ ~

19. GE Basics – Backward substitution

20. GE Basics – RHS updates

21. GE basics: summary (1) M x = b U x = yEquivalent system U: upper trg (2)Noticed that: Ly = bL: unit lower trg (3)U x = y LU x = b  M x = b GE  Efficient way of implementing GE: LU factorization

22. Solve M x = b Step 1 Step 2 Forward Elimination Solve L y = b Step 3 Backward Substitution Solve U x = y = M = L U Gaussian Elimination Basics Note: Changing RHS does not imply to recompute LU factorization

23. GE Basics – Fitting the pieces together

25. LU factorization Basics – Picture

26. LU Basics Source-row oriented approach algorithm For i = 1 to n-1 { “For each source row” For j = i+1 to n { “For each target row below the source” For k = i+1 to n { “For each row element beyond Pivot” } Pivot Multiplier

27. LU Basics Target-row oriented approach algorithm For i = 2 to n { “For each target row” For j = 1 to i-1 { “For each source row above the target” For k = j+1 to n { “For each row element beyond Pivot” } Pivot Multiplier

28. LU – Source-row and Target-row Multipliers Factored Portion Active Set k k Factored Portion Mult Active Set k k Source-Row oriented approach Target-Row oriented approach

29. For i = 1 to n-1 { “For each Row” For j = i+1 to n { “For each target Row below the source” For k = i+1 to n { “For each Row element beyond Pivot” } Pivot Multiplier multipliers Multiply-adds LU Basics – Computational Complexity

30. LU Basics – Limitations of the naïve approach Zero Pivots Small Pivots (Round-off error)  both can be solved with partial pivoting

31. At Step i Multipliers Factored Portion (L) Row i Row j What if Cannot form Simple Fix (Partial Pivoting) If Find Swap Row j with i LU Basics – Partial pivoting for zero pivots

32. Two Important Theorems 1) Partial pivoting (swapping rows) always succeeds if M is non singular 2) LU factorization applied to a diagonally dominant matrix will never produce a zero pivot LU Basics – Partial pivoting for zero pivots

33. LU Basics – Partial pivoting for small pivots GE

34. LU Basics – Partial pivoting for small pivots GE Rounded to 3 digits

35. 64 bits 52 bits 11 bits sign Double precision number Basic Problem Avoid sum and subtraction of large and tiny numbers  Avoid big multipliers An Aside on Floating Point Arithmetic LU Basics – Partial pivoting for small pivots

36. Partial Pivoting for Roundoff reduction LU Basics – Partial pivoting for small pivots Small Multipliers

37. LU Basics – Partial pivoting for small pivots GE Rounded to 3 digits swap

38. Pivoting strategies Partial Pivoting: –Only row interchange Complete Pivoting –Row and Column interchange Threshold Pivoting –Only if prospective pivot is found to be smaller than a certain threshold k k k k

39. Summary Existence and uniqueness review Gaussian elimination basics –GE basics –LU factorization –Pivoting