1. Lecture 11 Fundamental Theorems of Linear Algebra Orthogonalily and Projection Shang-Hua Teng

2. The Whole Picture Rank(A) = m = n Ax=b has unique solution Rank(A) = m < n Ax=b has n-m dimensional solution Rank(A) = n < m Ax=b has 0 or 1 solution Rank(A) < n, Rank(A) < m Ax=b has 0 or n-rank(A) dimensions

3. Basis and Dimension of a Vector Space A basis for a vector space is a sequence of vectors that –The vectors are linearly independent –The vectors span the space: every vector in the vector can be expressed as a linear combination of these vectors

4. Basis for 2D and n-D (1,0), (0,1) (1 1), (-1 –2) The vectors v 1,v 2,…v n are basis for R n if and only if they are columns of an n by n invertible matrix

5. Column and Row Subspace C(A): the space spanned by columns of A –Subspace in m dimensions –The pivot columns of A are a basis for its column space Row space: the space spanned by rows of A –Subspace in n dimensions –The row space of A is the same as the column space of A T, C(A T ) –The pivot rows of A are a basis for its row space –The pivot rows of its Echolon matrix R are a basis for its row space

6. Important Property I: Uniqueness of Combination The vectors v 1,v 2,…v n are basis for a vector space V, then for every vector v in V, there is a unique way to write v as a combination of v 1,v 2,…v n. v = a 1 v 1 + a 2 v 2 +…+ a n v n v = b 1 v 1 + b 2 v 2 +…+ b n v n So: 0=(a 1 - b 1 ) v 1 + (a 2 -b 2 )v 2 +…+ (a n -b n )v n

7. Important Property II: Dimension and Size of Basis If a vector space V has two set of bases –v 1,v 2,…v m. V = [v 1,v 2,…v m ] –w 1,w 2,…w n. W= [w 1,w 2,…w n ]. then m = n –Proof: assume n > m, write W = VA –A is m by n, so Ax = 0 has a non-zero solution –So VAx = 0 and Wx = 0 The dimension of a vector space is the number of vectors in every basis –Dimension of a vector space is well defined

8. Dimensions of the Four Subspaces Fundamental Theorem of Linear Algebra, Part I Row space: C(A T ) – dimension = rank(A) Column space: C(A)– dimension = rank(A) Nullspace: N(A) – dimension = n-rank(A) Left Nullspace: N(A T ) – dimension = m –rank(A)

9. Orthogonality and Orthogonal Subspaces Two vectors v and w are orthogonal if Two vector subspaces V and W are orthogonal if

10. Example: Orthogonal Subspace in 5 Dimensions The union of these two subspaces is R 5

11. Orthogonal Complement Suppose V is a vector subspace a vector space W The orthogonal complement of V is Orthogonal complement is itself a vector subspace

12. Dimensions of the Four Subspaces Fundamental Theorem of Linear Algebra, Part I Row space: C(A T ) – dimension = rank(A) Column space: C(A)– dimension = rank(A) Nullspace: N(A) – dimension = n-rank(A) Left Nullspace: N(A T ) – dimension = m –rank(A)

13. Orthogonality of the Four Subspaces Fundamental Theorem of Linear Algebra, Part II The nullspace is the orthogonal complement of the row space in R n The left Nullspace is the orthogonal complement of the column space in R m

14. Proof The nullspace is the orthogonal complement of the row space in R n

15. The Whole Picture C(A T ) N(A) RnRn RmRm C(A) N(A T ) xnxn A x n = 0 xrxr b A x r = b A x= b dim r dim n- r dim m- r

16. Uniqueness of The Typical Solution Every vector in the column space comes from one and only one vector x r from the row space Proof: suppose there are two x r, y r from the row space such that Ax r =A y r =b, then Ax r -A y r = A(x r -y r ) = 0 (x r -y r ) is in row space and nullspace hence must be 0 The matching of dim in row and column spaces

17. Deep Secret of Linear Algebra Pseudo-inverse Throw away the two null spaces, there is an r by r invertible matrix hiding insider A. In some sense, from the row space to the column space, A is invertible It maps an r-space in n space to an r-space in m-space

18. Invertible Matrices Any n linearly independent vector in R n must span R n. They are basis. So Ax = b is always uniquely solvable A is invertible

19. Projection Projection onto an axis (a,b) x axis is a vector subspace

20. Projection onto an Arbitrary Line Passing through 0 (a,b)

21. Projection on to a Plane

22. Projection onto a Subspace Input: 1. Given a vector subspace V in R m 2.A vector b in R m … Desirable Output: –A vector in x in V that is closest to b –The projection x of b in V –A vector x in V such that (b-x) is orthogonal to V

23. How to Describe a Vector Subspace V in R m If dim(V) = n, then V has n basis vectors –a 1, a 2, …, a n –They are independent V = C(A) where A = [a 1, a 2, …, a n ]

24. Projection onto a Subspace Input: 1. Given n independent vectors a 1, a 2, …, a n in R m 2.A vector b in R m … Desirable Output: –A vector in x in C([a 1, a 2, …, a n ]) that is closest to b –The projection x of b in C([a 1, a 2, …, a n ]) –A vector x in V such that (b-x) is orthogonal to C([a 1, a 2, …, a n ])

25. Think about this Picture C(A T ) N(A) RnRn RmRm C(A) N(A T ) xnxn A x n = 0 xrxr b A x r = b A x= b dim r dim n- r dim m- r