1. Multivariate Statistics Matrix Algebra II W. M. van der Veld University of Amsterdam

2. Overview The determinant of a matrix The matrix inverse System of equations

3. The determinant of a matrix The determinant of a matrix is a scalar and is denoted as |A| or det(A). Det(A) only exists when A is a square matrix. It has very important mathematical properties, but it is very difficult to provide a substantive definition. The determinant is necessary to compute the inverse of a matrix (A -1 ). –The inverse of a matrix is needed for solving systems of linear equations; multivariate statistics often comes down to this. –When the determinant is zero, there exists no solution to a system of linear equations. Let’s see how the value of the determinant is found.

4. The determinant of a matrix How to do it? The most simple case, a 2 by 2 matrix. Det(A)=|A|=? Cofactors

5. The determinant of a matrix One step further, a 3 by 3 matrix. Det(A)=|A|=? Cofactor

6. The determinant of a matrix You should have noted that for matrices larger than first order, computation of the determinant is a recursive process. This process stops each time a 1 by 1 determinant is encountered, and involves multiplication by the cofactors.

7. The determinant of a matrix Let A be a matrix of order n x n. If we omit one or more rows or columns from A, we obtain a matrix of smaller order, called a minor of the matrix. Similarly, we have minors of a determinant, and in particular, if we omit from the determinant the i th row and the j th column, the resulting minor will be square and its determinant will be symbolized |M ij |. This determinant is called a cofactor (c ij ) if we give it a sign equal to (-1) i+j, so that: c ij = (-1) i+j |M ij |. Using this notation we can write a formula for the expansion of a determinant of order n: In this version the determinant is expanded according to it’s i th row.

8. The determinant of a matrix The following rules are important for determinants, and can help you sometimes to simplify calculations: –The determinant of A has the same value as the determinant of A’. –The value of the determinant changes sign if one row (column) is interchanged with another row (column). –If a determinant has two equal rows (columns), its value is zero. –If a determinant has two rows (columns) with proportional elements, its value is zero. –If all elements in a row (column) are multiplied by a constant, the value of the determinant is multiplied by that constant. –If a determinant has a row (column) in which all elements are zero, the value of the determinant is zero. –The value of the determinant remains unchanged if one row (column) is added to or subtracted form another row (column). Moreover, if a row (column) is multiplied by a constant and then added to or subtracted from another row (column) the value remains unchanged.

9. The determinant of a matrix What is the determinant of:

10. The matrix inverse Let A be a square matrix. If we can find a matrix B of the same order as A such that AB=BA=I, then B is said to be the inverse of A and is symbolized A -1. A -1, if exists, can be found as follows. Let C be the matrix of cofactors of A (i.e., c ij is the cofactor obtained from the minor |M ij |); then Where C’ is the transpose of C (or if one prefers, C’ is the matrix of cofactors of A’). It is immediately seen that the inverse is undefined if A is not square (since then there is no determinant |A|), and also if |A| is equal to zero.

11. The matrix inverse Illustration that AA -1 = A -1 A = I.

12. The matrix inverse How did I get A -1 ? Compute determinant Now Compute C C transpose => C’ Calculate A -1

13. The matrix inverse Another way to calculate A -1. This way introduces you to solving systems of equations.

14. The matrix inverse Rules for algebra with inverse matrices: –AA -1 = A -1 A = I –(AB) -1 = B -1 A -1 –(ABC) -1 = C -1 B -1 A -1 Proof that (AB) -1 = B -1 A -1.

15. System of equations In the introduction I already mentioned that the basic linear equation y=bx will be very important for multivariate methods. Here we will discuss how to solve systems of such linear equations.

16. System of equations Illustration. Suppose we have the following set of equations: -3=1x 1 +4x 2 1=3x 1 +2x 2 The basic way to think about this problem set is finding the intersection, i.e. for which unknowns are the equations satisfied. This can be solved in a simple way (old style). The solution is basically the intersection of the lines represented by the equation. You won’t be surprised that there is a more general way to solve systems of linear equations, using matrix algebra.

17. System of equations We shall distinguish between homogeneous and nonhomogeneous equations. A homogeneous equation is of type a’x=0, –With a and x being vector of order nx1; a is a vector of coefficients, and x a vector of unknowns. –Eg., if a’=(2 -3) and x’=(x 1 x 2 ), then a’x=0 gives a homogeneous equation with two unknowns; 2x 1 -3x 2 =0. A nonhomogeneous equation is of type a’x=k, with k some constant (≠0). –Eg., 2x 1 -3x 2 =2.

18. System of equations Geometrically equations with two unknowns represent lines in a plane defined by the axes x 1 and x 2. Similarly, an equation with three unknowns, 3x 1 +3x 2 +5x 3 =4, represents a plane in an S 3 ; The space S 3 is defined by the three coordinates. Each triplet of values that satisfies the equation refers to a point located on the plane within S 3.

19. System of equations This can be generalized, so that an equation with n unknowns will stand for an (n-1) dimensional subspace S n. Such a subspace is called a hyperplane, symbolized by the letter V. So that V n-1 is a hyperplane in subspace S n., but S n will also of course contain the hyperplanes of a lower order. Recognize that principal component analyses is based upon this idea, except that the axis in the hyperplane are called factors or better principal components.

20. System of equations Rules for the intersection of hyperplanes –In general, two lines (V 2 ) will intersect in a point (V 1 ), except for parallel or coinciding lines. –Also two planes (V 3 ) will intersect in a line (V 2 ), except for similar exceptions. –The general rule is: in S n two hyperplanes V n-p and V n-q intersect in a V n-(p+q). In addition, when p+q=n, then the intersection is a point, when p+q>n then there is no intersection. What does the rule mean for equations? –That, in general there will be a unique solution only for n equations in n unknowns.

21. System of equations Solution for n nonhomogeneous equations with n unknowns What to do? Normally you divide by A so that you obtain a solution for x (give example: 15=3x). Matrix division is defined as multiplication by the inverse, so:

22. System of equations Solution for m equations with n unknowns: m=n. What to do? Normally you divide by A so that you obtain a solution for x (give example: 15=3x). Matrix division is defined as multiplication by the inverse, so:

23. System of equations Example. Suppose we have the following set of equations: -3=1x 1 +4x 2 1=3x 1 +2x 2 We already solved this one, resulting in x 1 =1 and x 2 =-1. The set of equations can be written as a matrix operation.

24. System of equations Thus, we have to find the inverse of: A => A -1 = C’/|A| We have to take the transpose of C

25. System of equations We have to divide by |A|. Thus the inverse matrix is.

26. System of equations Thus a solution for: -3=1x 1 +4x 2 1=3x 1 +2x 2 is found via

27. System of equations Solution for n-1 homogeneous equations with n unknowns There are many solutions. Why? Because: unknowns>knowns; or more x’s than n’s. The intersection (solution) therefore will represent a hyperplane, and all points in the hyperplane satisfy the equation. How to find this hyperplane?

28. System of equations How to find this hyperplane? One solution can be found when one of the unknowns is fixed to an arbitrary value k, with k≠0. The resulting set of equations are then solvable using: A -1 k=x. The solution then is x’ = (x 1 x 2 ….. x n =k). However, all vectors proportional to this vector will also satisfy the equation. Hence the general solution is: cx, with c an arbitrary constant, and x is column vector including the coordinates of the intersection. The solution includes c=0, because 0 is also a solution. This is very abstract, so….

29. System of equations An illustration. The problems set is: n-1 homogeneous equations in n unknowns. Two equations and three unknowns. 2x 1 -3x 2 +1x 3 =0; 4x 1 +1x 2 -2x 3 =0; Now fix x 3 to an arbitrary value k=1. Therefore: 2x 1 -3x 2 +1=0 => 2x 1 -3x 2 =-1; 4x 1 +1x 2 -2=0 => 4x 1 +1x 2 =2; Now there are two nonhomogeneous equations with two unknowns, which is solvable using: A -1 k=x:

30. System of equations Thus a solution will be: Now any vector proportional with this vector will satisfy the set of equations; e.g. x’=(5 8 14) All these vectors are considered one solution, and they form in this case a hyperplane V 2. Note that x 3 =1

31. System of equations Exercise, solve: x 1 + 2x 2 = 0 3x 1 + 7x 2 = 1 A -1 Ax = Ix = x = A -1 k So if Ax = k solve via x = A -1 k..... But it is not always so simple …

32. System of equations Sometimes, the requirement that m=n seems to be fulfilled, so that there should exist a solution. But consider the following cases. (Row 2 = 2 x Row 1) (Row 3 = Row 1 + Row 2) (Column 3 = Column 1 + Column 2), etc.

33. System of equations These situations are called linear dependence: –Given vectors: x 1, x 2,…, x n-1 –Another vector x n is linearly dependent if there exists constants α 1, α 2,…, α n-1 such that: x n = α 1 x 1 + α 2 x 2 + …+ α n-1 x n-1 Otherwise the vector x n is linearly independent. In case of linear dependence; |A|= 0. And then the inverse is not defined: A -1 =C’/|A|. And when the inverse is not defined we cannot find a solution via: A -1 k=x.

34. System of equations Generally a unique solution exists only if m=n, and |A|≠0 When are there ‘problems’? –If m<n there are many solutions, the problem is underdetermined. 8x 1 +10x 2 +14x 3 =9 4x 1 +12x 2 +16x 3 =10 –if m>n there are no solutions, the problem is overdetermined. 8x 1 +10x 2 =9 4x 1 +12x 2 =10 4x 1 +10x 2 =2

35. System of equations Using the idea of linear dependency, the rank of a matrix can be introduced. rank(A) = number of linearly independent rows or columns. Given an mxn matrix, with m ≥ n, then if –|A| ≠ 0  rank(A) = n  full rank, solvable –|A| = 0  rank(A) < n  rank deficient We will get back to the issue of rank.

36. Overdetermined Systems Minimization problem: Normal equations: (A’A)x = A’k Solution: x = (A’A) -1 A’k –A’A must be nonsingular; i.e. |A’A|≠0 –(A’A) -1 A’ is called the left inverse matrix Find Ax “closest” to k Least-squares distance measure

37. Underdetermined Systems Find “smallest” x that satisfies equations Minimum norm objective Constrained minimization problem: Solution: x = A’(AA’) -1 k –AA’ must be nonsingular –A’(AA’) -1 is called the right inverse