1. CS 395/495-26: Spring 2004 IBMR: Singular Value Decomposition (SVD Review) Jack Tumblin jet@cs.northwestern.edu

2. Matrix Multiply: A Change-of-Axes Matrix Multiply: Ax = bMatrix Multiply: Ax = b –x and b are column vectors –A has m rows, n columns x1x1……xnxnx1x1……xnxn… a 11 … a 1n …… a m1 … a mn = b1b1……bmbmb1b1……bmbm… x1x1x1x1 x2x2x2x2 b3b3b3b3 b2b2b2b2 b1b1b1b1 Ax=b x b Input (N dim) (2D) Output (M-dim) (3D)

3. Matrix Multiply: A Change-of-Axes Matrix Multiply: Ax = bMatrix Multiply: Ax = b –x and b are column vectors –A has m rows, n columns Matrix multiply is just a set of dot-products; it ‘changes coordinates’ for vector x, makes b.Matrix multiply is just a set of dot-products; it ‘changes coordinates’ for vector x, makes b. –Rows of A = A 1,A 2,A 3,... = new coordinate axes –Ax = a dot product for each new axis x1x1……xnxnx1x1……xnxn… a 11 … a 1n …… a m1 … a mn = b1b1……bmbmb1b1……bmbm… x1x1x1x1 x2x2x2x2 A2A2A2A2 A1A1A1A1 A3A3A3A3 b3b3b3b3 b2b2b2b2 b1b1b1b1 Ax=b x b Input (N dim) (2D) Output (M-dim) (3D)

4. Sphere of all unit-length x  ellipsoid of bSphere of all unit-length x  ellipsoid of b Output ellipsoid’s axes are always perpendicular (true for any ellipsoid)Output ellipsoid’s axes are always perpendicular (true for any ellipsoid) **THUS** we can make ┴, unit-length axes… How Does Matrix ‘stretch space’? x1x1x1x1 x2x2x2x2 b3b3b3b3 b2b2b2b2 b1b1b1b1 Ax=b 1 Input (N dim.) Output (M dim.) u1u1u1u1 u2u2u2u2 x 1 ellipsoid (disk) b

5. Sphere of all unit-length x  ellipsoid of bSphere of all unit-length x  ellipsoid of b Output ellipsoid’s axes form orthonormal basis vectors U i :Output ellipsoid’s axes form orthonormal basis vectors U i : Basis vectors U i are columns of U matrixBasis vectors U i are columns of U matrix SVD(A) = USV T columns of U = OUTPUT basis vectors SVD finds: ‘Output’ Vectors U, and… x1x1x1x1 x2x2x2x2 b3b3b3b3 b2b2b2b2 b1b1b1b1 Ax=b 1 Input (N dim.) Output (M dim.) u1u1u1u1 u2u2u2u2 x 1 ellipsoid (disk) b

6. SVD finds: …‘Input’ Vectors V, and… For each U i, make a matching V i that:For each U i, make a matching V i that: –Transforms to U i (with scaling s i ) : s i (A V i ) = U i –Forms an orthonormal basis of input space SVD(A) = USV T columns of V = INPUT basis vectors x1x1x1x1 x2x2x2x2 Ax=b x Input (N dim.) b3b3b3b3 b2b2b2b2 b1b1b1b1 Output (M dim.) u1u1u1u1 u2u2u2u2 v2v2v2v2 v1v1v1v1 b

7. SVD finds: …‘Scale’ factors S. We have Unit-length Output U i, vectors,We have Unit-length Output U i, vectors, Each from a Unit-length Input V i vector, soEach from a Unit-length Input V i vector, so So we need ‘scale factor’ (singular values) s i to link input to output: s i (A V i ) = U iSo we need ‘scale factor’ (singular values) s i to link input to output: s i (A V i ) = U i SVD(A) = USV T x1x1x1x1 x2x2x2x2 Ax=b x Input (N dim.) b3b3b3b3 b2b2b2b2 b1b1b1b1 Output (M dim.) u1u1u1u1 u2u2u2u2 v2v2v2v2 v1v1v1v1 s2s2s2s2 s1s1s1s1 b

8. SVD Review: What is it? Finish: SVD(A) = USV TFinish: SVD(A) = USV T –add ‘missing’ U i or V i, define these s i =0. –Singular matrix S: diagonals are s i : ‘v-to-u scale factors’ –Matrix U, Matrix V have columns U i and V i x1x1x1x1 x2x2x2x2 Ax=b x Input (N dim.) b3b3b3b3 b2b2b2b2 b1b1b1b1 Output (M dim.) u1u1u1u1 u3u3u3u3 u2u2u2u2 v2v2v2v2 v1v1v1v1 A = USV T ‘Find Input & Output Axes, Linked by Scale’ b

9. ‘Let SVDs Explain it All for You’ –cool! U and V are Orthonormal! U -1 = U T, V -1 = V T –Rank(A)? == # of non-zero singular values s i –‘ill conditioned’? Some s i are nearly zero –‘Invert’ a non-square matrix A ? Amazing! pseudo-inverse:A=USV T ; A VS -1 U T = I; so A -1 = V S -1 U T x1x1x1x1 x2x2x2x2 Ax=b x Input (N dim.) b3b3b3b3 b2b2b2b2 b1b1b1b1 Output (M dim.) u1u1u1u1 u3u3u3u3 u2u2u2u2 v2v2v2v2 v1v1v1v1 A = USV T ‘Find Input & Output Axes, Linked by Scale’

10. ‘Solve for the Null Space’ Ax=0 –Easy! x is the V i axis that doesn’t affect the output (s i = 0) –are all s i nonzero? then the only answer is x=0. –Null ‘space’ because V i is a DIMENSION--- x = Vi * a –More than 1 zero-valued s i ? null space >1 dimension... x = a V i1 + b V i2 + … x1x1x1x1 x2x2x2x2 Ax=b x Input (N dim.) b3b3b3b3 b2b2b2b2 b1b1b1b1 Output (M dim.) u1u1u1u1 u3u3u3u3 u2u2u2u2 v2v2v2v2 v1v1v1v1 A = USV T ‘Find Input & Output Axes, Linked by Scale’

11. More on SVDs: Handout Zisserman Appendix 3, pg 556 (cryptic)Zisserman Appendix 3, pg 556 (cryptic) Linear Algebra booksLinear Algebra books Good online introduction: (your handout) Leach, S. “Singular Value Decomposition – A primer,” Unpublished Manuscript, Department of Computer Science, Brown University. http://www.cs.brown.edu/research/ai/dynamics/tutorial/Postscript/Good online introduction: (your handout) Leach, S. “Singular Value Decomposition – A primer,” Unpublished Manuscript, Department of Computer Science, Brown University. http://www.cs.brown.edu/research/ai/dynamics/tutorial/Postscript/ http://www.cs.brown.edu/research/ai/dynamics/tutorial/Postscript/ Google on Singular Value Decomposition...Google on Singular Value Decomposition...

12. END