1. Extremum Properties of Orthogonal Quotients Matrices By Achiya Dax Hydrological Service, Jerusalem, Israel e-mail: dax20@water.gov.il

2. The Eckart – Young Theorem ( 1936 ) says that the “ truncated SVD” matrix A k = U k D k V k T solves the least norm problem minimize F ( B ) = || A - B || F 2 subject to rank ( B )  k ( Also called the Schmidt-Mirsky Theorem. )

3. Ky Fan’s Maximum Principle ( 1949 ) S a symmetric positive semi-definite n x n matrix Y k the set of orthogonal n x k matrices 1 + … + k = max { trace ( Y k T S Y k ) | Y k  Y k } Solution is obtained for the Spectral matrix V k = [v 1, v 2, …, v k ]. giving the sum of the largest k eigenvalues.

4. Outline * The Eckart – Young Theorem and Orthogonal Quotients matrices. * The Orthogonal Quotients Equality. * The Symmetric Quotients Equality and Ky Fan’s extremum principles. * An Extended Extremum Principle.

5. The Singular Value Decomposition A = U  V T  = diag {  1,  2, …,  p }, p = min { m, n } U = [u 1, u 2, …, u p ], U T U = I V = [v 1, v 2, …, v p ], V T V = I A V = U  A T U = V  A v j =  j u j, A T u j =  j v j j = 1, …, p.

6. Low - Rank Approximations A k = U k D k V k T D k = diag {  1,  2, …,  k }, U k = [u 1, u 2, …, u k ], U k T U k = I V k = [v 1, v 2, …, v k ], V k T V k = I

7. The Eckart – Young Theorem ( 1936 ) says that the “truncated SVD” matrix A k = U k D k V k T solves the least norm problem minimize F ( B ) = || A - B || F 2 subject to rank ( B )  k ( Also called the Schmidt-Mirsky Theorem. )

8. Rank - k Matrices B = X k R k Y k T where R k is a k x k matrix X k = [x 1, x 2, …, x k ], X k T X k = I Y k = [y 1, y 2, …, y k ], Y k T Y k = I

9. The Eckart – Young Problem can be rewritten as minimize F ( B ) = || A - X k R k Y k T || F 2 subject to X k T X k = I and Y k T Y k = I.

10. Theorem 1 : Given a pair of orthogonal matrices, X k and Y k, the related “Orthogonal Quotients Matrix” X k T A Y k = ( x i T Ay j ) solves the problem minimize F ( R k ) = || A - X k R k Y k T || F 2

11. Notation: X k - denotes the set of all real m x k orthogonal matrices X k, X k = [x 1, x 2, …, x k ], X k T X k = I Y k - denotes the set of all real n x k orthogonal matrices Y k, Y k = [y 1, y 2, …, y k ], Y k T Y k = I

12. Corollary 1 : The Eckart – Young Problem can be rewritten as minimize F ( X k, Y k ) = || A - X k R k Y k T || F 2 subject to X k  X k and Y k  Y k, where R k is the “Orthogonal Quotients Matrix” R k = X k T A Y k.

13. The Orthogonal Quotients Equality For any pair of orthogonal matrices, X k  X k and Y k  Y k, || A - X k R k Y k T || F 2 = || A || F 2 - || R k || F 2 where R k is the orthogonal quotients matrix R k = X k T A Y k.

14. Corollary : The Eckart – Young Problem minimize F ( X k, Y k ) = || A - X k R k Y k T || F 2 subject to X k  X k and Y k  Y k. is equivalent to maximize || X k T A Y k || F 2 subject to X k  X k and Y k  Y k, and the SVD matrices U k and V k solves both problems, giving the optimal value of  1 2 +  2 2 + … +  k 2.

15. Question : Is the related minimum problem minimize || (X k ) T A Y k || F 2 subject to X k  X k and Y k  Y k, solvable ?

16. Using the Orthogonal Quotients Equality the last problem takes the form maximize F ( X k, Y k ) = || A - X k R k Y k T || F 2 subject to X k  X k and Y k  Y k.

17. Note that the more general problem maximize F ( B ) = || A - B || F 2 subject to rank ( B )  k is not solvable.

18. Returning to symmetric matrices How we extend the Orthogonal Quotients Equality to symmetric matrices ?

19. The Spectral Decomposition S = ( S ij ) a symmetric positive semi-definite n x n matrix With eigenvalues     n  and eigenvectors v 1, v 2, …, v n S v j = j v j, j = 1, …, n. S V = V D V = [v 1, v 2, …, v n ], V T V = V V T = I D = diag { 1, 2, …, n } S = V D V T =  j v j v j T

20. Recall that Rayleigh Quotient Matrices S k = Y k T S Y k play important role in Ky Fan’s Extremum Principles.

21. Ky Fan’s Maximum Principle S a symmetric positive semi-definite n x n matrix Y k the set of orthogonal n x k matrices 1 + … + k = max { trace ( Y k T S Y k ) | Y k  Y k } Solution is obtained for the Spectral matrix V k = [v 1, v 2, …, v k ]. which is related to the largest k eigenvalues.

22. Ky Fan’s Minimum Principle S a symmetric n x n matrix. Y k the set of orthogonal n x k matrices. n - k+1 + … + n = min { trace ( Y k T S Y k ) | Y k  Y k } Solution is obtained for the Spectral matrix V k = [v n-k+1, …, v n ], which is related to the smallest k eigenvalues.

23. Question : Can we formulate the Symmetric Quotients Equality in terms of trace ( S k ) = trace ( Y k T S Y k ) =  y j T S y j

24. The Symmetric Quotients Equality S a symmetric n x n matrix Y k  Y k an orthogonal n x k matrix S k = Y k T S Y k the related “Rayleigh quotient matrix” trace ( S - Y k S k Y k T ) = trace ( S ) - trace( S k )

25. Corollary 1 : Ky Fan’s maximum problem maximize trace (Y k S Y k T ) subject to Y k  Y k, is equivalent to minimize trace ( S - Y k S k Y k T ) subject to Y k  Y k. The Spectral matrix V k = [v 1, v 2, …, v k ] solves both problems, giving the optimal value of 1 + … + k.

26. Recall that : The Eckart – Young Problem minimize F ( X k, Y k ) = || A - X k R k Y k T || F 2 subject to X k  X k and Y k  Y k. is equivalent to maximize || X k T A Y k || F 2 subject to X k  X k and Y k  Y k, and the SVD matrices U k and V k solves both problems, giving the optimal value of  1 2 +  2 2 + … +  k 2.

27. Corollary 2 : Ky Fan’s minimum problem minimize trace (Y k S Y k T ) subject to Y k  Y k, is equivalent to maximize trace ( S - Y k T S k Y k ) subject to Y k  Y k. The matrix V k = [v n-k+1, …, v n ] solves both problems, giving the optimal value of n-k+1 + … + n.

28. Extended Exremum Principles Can we extend these extremums from eigenvalues of symmetric matrices to singular values of rectangular matrices ?

29. Notation: 1  m*  m, 1  n*  n, X m* - denotes the set of all real m x m* orthogonal matrices X m*, X m* = [x 1, x 2, …, x m* ], X m* T X m* = I Y n* - denotes the set of all real n x n* orthogonal matrices Y n*, Y n* = [y 1, y 2, …, y n* ], Y n* T Y * = I

30. Notations : Given X m*  X m* and Y n*  Y m*, the m* x n* matrix ( X m* ) T A Y n* = ( x i T Ay j ) is called “ Orthogonal Quotients Matrix”.

31. Notations : The singular values of the Orthogonal Quotients Matrix ( X m* ) T A Y n* = ( x i T Ay j ) are denoted as  1   2  …   k  0, where k = min { m*, n* }.

32. Questions : Which choice of orthogonal matrices X m*  X m* and Y n*  Y m*, maximizes (or minimizes ) the sum (  1 ) p + (  2 ) p + … + (  k ) p where p > 0 is a given positive constant.

33. An Extended Maximum Principle : The SVD matrices U m* = [ u 1, u 2, …, u m* ] and V n* = [v 1, v 2, …, v n* ] solve the problem maximize F ( X m*, Y n* ) = (  1 ) p + (  2 ) p + … + (  k ) p subject to X m*  X m* and Y n*  Y n*, for any positive power p > 0, giving the optimal value of (  1 ) p + (  2 ) p + … + (  k ) p.

34. An Extended Maximum Principle That is, for any positive power p > 0, (  1 ) p + (  2 ) p + … + (  k ) p = max{ (  1 ) p + (  2 ) p +…+ (  k ) p | X m*  X m* and Y n*  Y n* }, and the maximal value is attained for the matrices U m* = [u 1, u 2, …, u m* ] and V n* = [v 1, v 2, …, v n * ].

35. The proof is based on “rectangular” versions of Cauchy Interlace Theorem and Poincare Separation Theorem.

36. Corollary 1 : When p = 1 the SVD matrices U m* = [ u 1, u 2, …, u m* ] and V n* = [v 1, v 2, …, v n* ] solve the “Rectangular Ky Fan problem” maximize F ( X m*, Y n* ) =  1 +  2 + … +  k subject to X m*  X m* and Y n*  Y n*, giving the optimal value of  1 +  2 + … +  k.

37. Corollary 2 : When p = 1 and m* = n* = k the SVD matrices U k = [ u 1, u 2, …, u k ] and V k = [v 1, v 2, …, v k ] solve the maximum trace problem maximize F ( X k, Y k ) = trace ( (X k ) T A Y k ) subject to X k  X k and Y k  Y k, giving the optimal value of  1 +  2 + … +  k. * See also Horn & Johnson, “Topics in Matrix Analysis”, p. 195.

38. Corollary 3 : When p = 2 the SVD matrices U m* = [ u 1, u 2, …, u m* ] and V n* = [v 1, v 2, …, v n* ] solve the “rectangular Eckart – Young problem” maximize F ( X m*, Y n* ) = || (X m* ) T A Y n* || F 2 subject to X m*  X m* and Y n*  Y n*, giving the optimal value of (  1 ) 2 + (  2 ) 2 + … + (  k ) 2.

39. An Extended Minimum Principle Question : Can we prove a similar minimum principle ? Answer : Yes, but the solution matrices are more complicated.

40. An Extended Minimum Principle : Here we consider the problem minimize F( X m*, Y n* ) = (  1 ) p + (  2 ) p + … + (  k ) p subject to X m*  X m* and Y n*  Y n*, for any positive power p > 0. The solution matrices are obtained by deleting some columns from the SVD matrices U = [u 1, u 2, …, u m ] and V = [v 1, v 2, …, v n ].

41. Summary * The Eckart – Young Theorem and Orthogonal Quotients matrices. * The Orthogonal Quotients Equality. * The Symmetric Quotients Equality and Ky Fan’s extremum principles. * An Extended Extremum Principle.

42. The END Thank You