1. QR Factorization –Direct Method to solve linear systems Problems that generate Singular matrices –Modified Gram-Schmidt Algorithm –QR Pivoting Matrix must be singular, move zero column to end. –Minimization view point  Link to Iterative Non stationary Methods (Krylov Subspace) Outline

2. 1 1 v1 v2v3 v4 The resulting nodal matrix is SINGULAR, but a solution exists! LU Factorization fails Singular Example

3. The resulting nodal matrix is SINGULAR, but a solution exists! Solution (from picture): v 4 = -1 v 3 = -2 v 2 = anything you want  solutions v 1 = v 2 - 1 LU Factorization fails Singular Example One step GE

4. Recall weighted sum of columns view of systems of equations M is singular but b is in the span of the columns of M QR Factorization Singular Example

5. Orthogonal columns implies : Multiplying the weighted columns equation by i-th column: Simplifying using orthogonality : QR Factorization – Key idea If M has orthogonal columns

6. Picture for the two-dimensional case Non-orthogonal Case Orthogonal Case QR Factorization - M orthonormal M is orthonormal if:

7. How to perform the conversion? QR Factorization – Key idea

8. QR Factorization Projection formula

9. Formulas simplify if we normalize QR Factorization – Normalization

10. Mx=b  Qy=b  Mx=Qy QR Factorization – 2 x 2 case

11. Two Step Solve Given QR QR Factorization – 2 x 2 case

12. To Insure the third column is orthogonal QR Factorization – General case

13. In general, must solve NxN dense linear system for coefficients

14. To Orthogonalize the Nth Vector QR Factorization – General case

15. To Insure the third column is orthogonal QR Factorization – General case Modified Gram-Schmidt Algorithm

16. For i = 1 to N “For each Source Column”   For j = i+1 to N { “For each target Column right of source”   end  end Normalize QR Factorization Modified Gram-Schmidt Algorithm (Source-column oriented approach)

17. QR Factorization – By picture

18. Suppose only matrix-vector products were available? More convenient to use another approach QR Factorization – Matrix-Vector Product View

19. For i = 1 to N “For each Target Column” For j = 1 to i-1 “For each Source Column left of target” end Normalize QR Factorization Modified Gram-Schmidt Algorithm (Target-column oriented approach)

20. QR Factorization r 11 r 12 r 22 r 13 r 23 r 33 r 14 r 24 r 34 r 44 r 11 r 22 r 12 r 14 r 13 r 23 r 24 r 33 r 34 r 44

21. What if a Column becomes Zero? Matrix MUST BE Singular! 1)Do not try to normalize the column. 2)Do not use the column as a source for orthogonalization. 3) Perform backward substitution as well as possible QR Factorization – Zero Column

22. Resulting QR Factorization QR Factorization – Zero Column

23. Recall weighted sum of columns view of systems of equations M is singular but b is in the span of the columns of M QR Factorization – Zero Column

24. Reasons for QR Factorization QR factorization to solve Mx=b –Mx=b  QRx=b  Rx=Q T b where Q is orthogonal, R is upper trg O(N 3 ) as GE Nice for singular matrices –Least-Squares problem Mx=b where M: mxn and m>n Pointer to Krylov-Subspace Methods –through minimization point of view

25. Minimization More General! QR Factorization – Minimization View

26. One dimensional Minimization Normalization QR Factorization – Minimization View One-Dimensional Minimization

27. One dimensional minimization yields same result as projection on the column! QR Factorization – Minimization View One-Dimensional Minimization: Picture

28. Residual Minimization Coupling Term QR Factorization – Minimization View Two-Dimensional Minimization

29. QR Factorization – Minimization View Two-Dimensional Minimization: Residual Minimization Coupling Term To eliminate coupling term: we change search directions !!!

30. More General Search Directions Coupling Term QR Factorization – Minimization View Two-Dimensional Minimization

31. More General Search Directions QR Factorization – Minimization View Two-Dimensional Minimization Goal : find a set of search directions such that In this case minimization decouples !!! p i and p j are called M T M orthogonal

32. i-th search direction equals    orthogonalized unit vector Use previous orthogonalized Search directions QR Factorization – Minimization View Forming M T M orthogonal Minimization Directions

33. QR Factorization – Minimization View Minimizing in the Search Direction When search directions p j are M T M orthogonal, residual minimization becomes:

34. For i = 1 to N “For each Target Column” For j = 1 to i-1 “For each Source Column left of target” end Normalize Orthogonalize Search Direction QR Factorization – Minimization View Minimization Algorithm

35. Intuitive summary QR factorization  Minimization view (Direct)(Iterative) Compose vector x along search directions: –Direct: composition along Q i (orthonormalized columns of M)  need to factorize M –Iterative: composition along certain search directions  you can stop half way About the search directions: –Chosen so that it is easy to do the minimization (decoupling)  p j are M T M orthogonal –Each step: try to minimize the residual

36. MMM M T M Orthonormal Compare Minimization and QR

37. Summary Iterative Methods Overview –Stationary –Non Stationary QR factorization to solve Mx=b –Modified Gram-Schmidt Algorithm –QR Pivoting –Minimization View of QR Basic Minimization approach Orthogonalized Search Directions Pointer to Krylov Subspace Methods

38. Forward Difference Formula Of order o(h 2 ) for Numerical Differentiation Y’ = e -x sin(x)

39. Terima kasih