Lecture XXVI

 The material for this lecture is found in James R. Schott Matrix Analysis for Statistics (New York: John Wiley & Sons, Inc. 1997).  A matrix A of size m x n is an m x n rectangular array of scalars:

 It is sometimes useful to partition matrices into vectors.

 The sum of two identically dimensioned matrices can be expressed as

 In order to multiply a matrix by a scalar, multiply each element of the matrix by the scalar.  In order to discuss matrix multiplication, we first discuss vector multiplication. Two vectors x and y can be multiplied together to form z (z=x y) only if they are conformable. If x is of order 1 x n and y is of order n x 1, then the vectors are conformable and the multiplication becomes:

 Extending this discussion to matrices, two matrices A and B can be multiplied if they are conformable. If A is order k x n and B is of order n x l. then the matrices are conformable. Using the partitioned matrix above, we have

 Theorem 1.1 Let  and  be scalars and A, B, and C be matrices. Then when the operations involved are defined, the following properties hold: ◦ A+B=B+A. ◦ (A+B)+C=A+(B+C). ◦ (A+B)=  A+  B. ◦ (  +  )A=  A+  A. ◦ A-A=A+(-A)=(0).

◦ A(B+C)=AB+AC. ◦ (A+B)C=AC+BC. ◦ (AB)C=A(BC).  The transpose of an m x n matrix is a n x m matrix with the rows and columns interchanged. The transpose of A is denoted A’.

 Theorem 1.2 Let  and  be scalars and A and B be matrices. Then when defined, the following hold ◦ (  A)’=  A’. ◦ (A’)’=A. ◦ (  A+  B)’=  A’+  B’. ◦ (AB)’=B’A’.

 The trace is a function defined as the sum of the diagonal elements of a square matrix.

 Theorem 1.3 Let  be scalar and A and B be matrices. Then when the appropriate operations are defined, we have ◦ tr(A’)=tr(A). ◦ tr(  A)=  tr(A). ◦ tr(A+B)=tr(A)+tr(B). ◦ tr(AB)=tr(BA). ◦ tr(A’A)=0 if and only if A=(0).

 Traces can be very useful in statistical applications. For example, natural logarithm of the normal distribution function can be written as: ◦ Jan R. Magnus and Heinz Neudecker Matrix Differential Calculus with Applications in Statistics and Econometrics (New York: John Wiley & Sons, 1988) p. 314.

 The Determinant is another function of square matrices. In its most technical form, the determinant is defined as: where the summation is taken over all permutations, (i 1,i 2,…i m ) of the set of integers (1,…m), and the function f(i 1,i 2,…i m ) equals the number of transpositions necessary to change (i 1,i 2,…i m ).

 In the simple case of a 2 x 2, we have two possibilities (1,2) and (2,1). The second requires one transposition. Under the basic definition of the determinant:

 In the slightly more complicated case of a 3 x 3, we have six possibilities (1,2,3), (2,1,3), (2,3,1), (3,2,1), (3,1,2), (1,3,2). Each one of these differs from the previous one by one transposition. Thus, the number of transpositions are 0, 1, 2, 3, 4, 5. The determinant is then defined as:

 A more straightforward definition involves the expansion down a column or across the row. ◦ In order to do this, I want to introduce the concept of principal minors.  The principal minor of an element in a matrix is the matrix with the row and column of the element removed.  The determinant of the principal minor times negative one raised to the row number plus the column number is called the cofactor of the element.

◦ The determinant is then the sum of the cofactors times the elements down a particular column or across the row:

 In the three by three case:

 Theorem 1.4 If  is a scalar and A is an m x m matrix, then the following properties hold: ◦ |A’|=|A|. ◦ |  A|=  m |A|. ◦ If A is a diagonal matrix, then |A|=a 11 a 22 …a mm. ◦ If all elements of a row (or column) of A are zero, |A|=0. ◦ If two rows (or columns) of A are proportional to one another, |A|=0.

◦ The interchange of two rows (or columns) of A changes the sign of |A|. ◦ If all the elements of a row (or column) of A are multiplied by , then the determinant is multiplied by . ◦ The determinant of A is unchanged when a multiple of one row (or column) is added to another row (or column).

 Any m x m matrix A such that |A|≠0 is said to be a nonsingular matrix and possesses an inverse denoted A -1.

 Theorem 1.6 If  is a nonzero scalar, and A and B are nonsingular m x m matrices, then ◦ (  A) -1 =  -1 A -1. ◦ (A’) -1 =(A -1 )’. ◦ (A -1 ) -1 =A. ◦ |A -1 |=|A| -1. ◦ If A=diag(a 11,…a mm ), then A -1 =diag(a 11 -1,…a mm -1 ). ◦ If A=A’, then A -1 =(A -1 )’. ◦ (AB) -1 =B -1 A -1.

 The most general definition of an inverse involves the adjoint matrix (denoted A # ). The adjoint matrix of A is the transpose of the matrix of cofactors of A. By construction of the adjoint, we know that:

 In order to see this identity, note that  Focusing on the first point

 Given this expression, we see that

 The rank of a matrix is the number of linearly independent rows or columns. One way to determine the rank of any general matrix m x n is to delete rows or columns until the resulting r x r matrix has a nonzero determinant. What is the rank of the above matrix? If the above matrix had been:

note |A|=0. Thus, to determine the rank, we delete the last row and column leaving

 The rank of a matrix A remains unchanged by any of the following operations, called elementary transformations: ◦ The interchange of two rows (or columns) of A. ◦ The multiplication of a row (or column) of A by a nonzero scalar. ◦ The addition of a scalar multiple of a row (or column) of A to another row (or column) of A.

 An m x 1 vector p is said to be a normalized vector or a unit vector if p’p=1. The m x 1 vectors p 1, p 2,…p n where n is less than or equal to m are said to be orthogonal if p i ’p j =0 for all i not equal to j. If a group of n orthogonal vectors are also normalized, the vectors are said to be orthonormal. An m x m matrix consisting of orthonormal vectors is said to be orthogonal. It then follows:

 It is possible to show that the determinant of an orthogonal matrix is either 1 or –1.

 In general, the a quadratic form of a matrix can be written as:  We are most often interested in the quadratic form x’Ax.

 Every symmetric matrix A can be classified into one of five categories: ◦ If x’Ax > 0 for all x ≠ 0, the A is positive definite. ◦ If x’Ax ≥ 0 for all x ≠ 0 and x’Ax=0 for some x ≠ 0, the A is positive semidefinite. ◦ If x’Ax < 0 for all x ≠ 0 then A is negative definite. ◦ If x’Ax ≤ 0 for all x ≠ 0 and x’Ax=0 for some x ≠ 0, the A is negative semidefinite. ◦ If x’Ax>0 for some x and x’Ax<0 for some x, then A is indefinite.

 Definition 2.1. Let S be a collection of m x 1 vectors satisfying the following: ◦ If x 1 ε S and x 2 ε S, then x 1 +x 2 ε S. ◦ If x ε S and  is a real scalar, the  x ε S. Then S is called a vector space in m- dimensional space. If S is a subset of T, which is another vector space in m- dimensional space, the S is called a vector subspace of T.

 Definition 2.2 Let {x 1,…x n } be a set of m x 1 vectors in the vector space S. If each vector in S can be expressed as a linear combination of the vectors x 1,…x n, then the set {x 1,…x n } is said to span or generate the vector space S, and {x 1,…x n } is called a spanning set of S.

 Definition 2.6 The set of m x 1 vectors {x 1,…x n } is said to be a linearly independent if the only solution to the equation is the zero vector  1 =…  n =0.

 This reduction implies that: Or that the third column of the matrix is a linear combination of the first two.