1. Introduction to Matrices Douglas N. Greve greve@nmr.mgh.harvard.edu

2. Simplifies notation Simplifies concepts Grounding for general linear model (GLM) Simplifies implementation (eg, matlab, octave) Optimization Why Matrices?

3. A matrix is a table of numbers, like a spread sheet Each entry is called an “element” There is a value in each element (no empty cells) Elements can be whole numbers, positive, negative, fractions, real, imaginary, symbolic variables What is the matrix? 24y  3.3i-7.4 M = M M

4. Parts of Matrices Size/Dimensions Elements Groupings of Elements Diagonals Triangles Vectors

5. Number of Rows Number of Columns Rows-by-Columns (RowsxCols) Matrix Size/Dimension 2x3 24y  3.3i-7.4 1x3 24y 2  2x1  1x1

6. Scalar is just a number or value No dimension or size Scalar versus Matrix  = -1 No brackets No dimension Not a 1x1  = [-1] 1x1

7. Indicated by its row and column number Scalar value Matrix Element 2x3 (1,1) (1,2) (1,3)  (2,2) (2,3) M = M(2,3) = -7.4 M(i,j) M(row,col) M ij 24y  3.3i-7.4

8. Row = Column + Offset Diagonals 246 135 789 3x3 246 135 789 Row = Column + 0 (Main Diagonal) Row = Column - 1 (First Upper Diagonal) Row = Column - 2 (Second Upper Diagonal) Row = Column - 1 (First Lower Diagonal) Row = Column - 2 (Second Lower Diagonal) } Off-Diagonals }

9. Triangles 246 135 789 3x3 Upper Triangle Lower Triangle

10. Special Type of Matrix Number of Rows or Columns = 1 Often given lower-case variable names Vector 1x3 Row Vector 2  24y 2x1 Column Vector

11. A matrix consists of vectors 24x  3.3i-7.4 24x  3.3i-7.4 24y  3.3i-7.4 Two 1x3 row vectorsThree 2x1 column vectors 2x3

12. Special Matrices Square Diagonal Identity Triangular Symmetric Toeplitz

13. Rows = Columns Square Matrix 2x2 24  3.3i  1x1

14. All off-diagonal elements = 0 M(i,j)=0 if i != j Diagonal Matrix 3x3 200 030 009

15. Main diagonal has all 1s All off-diagonal elements = 0 Square Symbol “ I” Identity Matrix 3x3 100 010 001 I =I = Any matrix multiplied by the identity is itself

16. Triangular Matrices 246 035 009 3x3 Upper Triangular – all values below the main diagonal are 0 200 130 789 3x3 Lower Triangular – all values above the main diagonal are 0

17. Symmetric Matrix 2 46 4 3 5 65 9 3x3 M(row,col) = M(col,row) Reflect across main diagonal Square

18. All elements on a diagonal are the same Toeplitz Matrix 246 124 712 3x3 24 12 71 3x2 A causal, time invariant linear system is represented by an upper triangular Toeplitz matrix

19. All elements are 1 Any number of rows or columns Ones Matrix 2x3 111 1 11 1x3 111 1 1 2x1  1x1

20. All elements are 0 Any number of rows or columns Zeros Matrix 2x3 000 0 00 1x3 000 0 0 2x1  1x1

21. Matrix Operations Addition/Subtraction Vector Multiplication Matrix Multiplication Multiplication by scalar Transpose Trace Inverse Pseudo Inverse

22. Element-by-Element Addition/Subtraction Size of two matrices must be the same C(row,col) = A(row,col) +/- B(row,col) Matrix Addition/Subtraction 246 135 A = 07 9 1-23.3 B = 21115 218.3 C = 2x3 C = A + B (your first matrix equation!)

23. One Row Vector and One Column Vector Same length, Results in 1x1 Inner Product, Dot Product, “Scalar” Product Like a correlation “Multiply and accumulate” Vector Multiplication 246 x = 0 7 1 y = 1x3 3x1 z = x.y= x*y =  x i *y i 2 4 6 * * * 0 1 7 0+4+42 = 46

24. Series of vector multiplications Same “inner dimension” Matrix Multiplication 246 135 A = 0 1 B = 2x32x3 3x43x4 3 4 5 7 2 6 9 8 2 3 c ij =Arow i *Bcol j c 11 2x42x4 C=A*B = c 12 c 13 c 14 c 21 c 22 c 23 c 24 Dimensions: (2x3) *(3x4)= (2x4 )

25. Series of vector multiplications Same “inner dimension” Matrix Multiplication 246 135 A = 0 1 B = 2x32x3 3x43x4 3 4 5 7 2 6 9 8 2 3 c11c11 2x42x4 C=A*B = c 12 c 13 c 14 c 21 c 22 c 23 c 24 2460 1 3 c11=c11= 2*0+4*1+6*3= 22

26. Series of vector multiplications Same “inner dimension” Matrix Multiplication 246 135 A = 0 1 B = 2x32x3 3x43x4 3 4 5 7 2 6 9 8 2 3 22 2x42x4 C=A*B = c12c12 c 13 c 14 c 21 c 22 c 23 c 24 2464 5 7 c12=c12= 2*4+4*5+6*7= 70

27. Series of vector multiplications Same “inner dimension” Matrix Multiplication 246 135 A = 0 1 B = 2x32x3 3x43x4 3 4 5 7 2 6 9 8 2 3 22 2x42x4 C=A*B = 70c 14 c 21 c 22 c 23 c 24 2462 6 9 c13=c13= 2*2+4*6+6*9= 82 c13c13

28. Series of vector multiplications Same “inner dimension” Matrix Multiplication 246 135 A = 0 1 B = 2x32x3 3x43x4 3 4 5 7 2 6 9 8 2 3 22 2x42x4 C=A*B = 70c14c14 c 21 c 22 c 23 c 24 2468 2 3 c14=c14= 2*8+4*2+6*3= 42 82

29. Series of vector multiplications Same “inner dimension” Matrix Multiplication 246 135 A = 0 1 B = 2x32x3 3x43x4 3 4 5 7 2 6 9 8 2 3 22 2x42x4 C=A*B = 7042 c21c21 c 22 c 23 c 24 1350 1 3 c21=c21= 1*0+4*1+5*3= 18 82

30. Series of vector multiplications Same “inner dimension” Matrix Multiplication 246 135 A = 0 1 B = 2x32x3 3x43x4 3 4 5 7 2 6 9 8 2 3 22 2x42x4 C=A*B = 7042 18c22c22 c 23 c 24 1354 5 7 c22=c22= 1*4+4*5+5*7= 54 82

31. One Row Vector one Column Vector Same length (N) Result is NxN matrix Special case of Matrix Multiplication Outer Product (Vector Multiplication) 246 x = 0 7 1 y = 1x3 3x1 z = y*x 0 7 1 246 1x3 3x1 0 14 3x3 00 246 2842

32. Multiply each element by scalar value Multiplication with a scalar 246 135 A =  = -1 2x3  = No brackets No dimension Not a 1x1 -2-4-6 -3-5 2x3

33. “Reflect” across diagonal B(row,col) = A(col,row) Matrix Transpose 246 135 A = 2 4 B=A T =A t =A’= 2x3 3x2 6 1 3 5 If A=A ’, then A is symmetric (must be square)

34. Sum of diagonal Elements Trace =  M(i,i) Trace 246 135 789 M = Trace = 2 + 3 + 9 = 14

35. Determinant c 11 2x2 C = c 12 c 21 c 22  = c 11 * c 22 - c 12 * c 21 Property of a square matrix Complicated in general Simple for a 2x2

36. C*A = I, then A= C -1 Must be square Complicated in general Simple for a 2x2 Matrix Inverse c 11 2x2 C = c 12 c 21 c 22  = c 11 * c 22 - c 12 * c 21 c 22 2x2 C -1 = -c 12 -c 21 c 11 

37. Invertibility 1.0 2x2 C = 2.0 0.51.0  = 1.0*1.0 - 2.0*0.5 = 1-1 = 0 1.0 2x2 C -1 = -2.0 -0.51.0  IMPORTANT!!! Not all matrices are invertible  =0 “Singular”

38. Singularity and “Ill-Conditioned” 1.0 2x2 C = 2.0 0.51.0  = 1.0*1.0 - 2.0*0.5 = 1-1 = 0 Column 2 = twice Column 1 Linear Dependence Ill-Conditioned:  is “close” to 0 Relates to efficiency of a GLM.

39. Can be used for non-square Same rules on invertibility apply Least-mean-square (LMS) for over-determined system Pseudo-Inverse If X has more columns than rows X + = (X’ * X) -1 * X’ Check: X + * X = (X’ * X) -1 * X’ * X=I If X has more rows than columns X + = X’ * (X * X’) -1 * Check: X * X + = X * X’ * (X * X’) -1 =I

40. Matrix Applications

41. Sum/Length/Average of a List length(y) = x ’ *x = 1*1 + 1*1 + 1*1 = 3 sum(y) = x ’ *y = 1*1 + 1*2 + 1*3 = 6 x = 3x1 1 1 1 1x3 x ’ = [1 1 1] 1x3 “Ones vector” 3x1 Average  = (x ’ *x) -1 (x’*y) [Note: GLM] = (3 -1 )*6 = 6/3 = 2 y = 3x1 1 2 3

42. Selective Average length(y) = x ’ *x = 0*0+ 1*1 + 1*1 = 2 sum(y) = x ’ *y = 0*1 + 1*2 + 1*3 = 5 x = 3x1 0 1 1 1x3 x ’ = [0 1 1] 1x3 “Ones vector” 3x1 Average  = (x ’ *x) -1 (x’*y) = (2 -1 )*5 = 5/2 = 2.5 y = 3x1 1 2 3

43. Sum of the Squares of a List y’ = [1 2 3] 1x3 SS(y) = y ’ *y = 1*1 + 2*2 + 3*3 = 14 y = 3x1 1 2 3 SS =  y 2 )

44. Variance of a List Residual e = y –  *x  2 = sqrt(e’*e)/DOF DOF = x’*x - 1 = #RowsX-#ColsX y = 3x1 1 2 3 x = 3x1 1 1 1  2 =  (y-  ) 2 (N-1)

45. Vector product is 0 Property of two vectors Uncorrelated/Independence - statistical property Orthogonality 15 x = 2.2 3 y = 1x3 3x1 z = x*y =  x i *y i 1 * 2 + -1 * 3 + 5 *.2 = [0] 1x1

46. 46 Fitting a Line HRF Amplitude IQ, Height, Weight Independent Variable Dependent Variable, Measurement x1x2 y2 y1 Subject 1 Subject 2 Thickness Age

47. 47 Linear Model Intercept: b Slope: m Age x1x2 y2 y1 System of Linear Equations y1 = 1*b + x1*m y2 = 1*b + x2*m Two Eqns, Two Unknowns “Intercept” = “Offset” m= (y2-y1) (x2-x1) b= (x2*y1-x1*y2) (x2-x1) 1.Solve 1 st Eq for b 2.Substitute into 2 nd Eq 3.Solve 2 nd equation for m 4.Substitute m back into Eq for b

48. 48 Matrix Formulation/GLM Intercept: b Slope: m Age x1x2 y2 y1 y1 = 1*b + x1*m y2 = 1*b + x2*m  - parameter vector X - design matrix y1 y2 1 x1 1 x2 bmbm =* y = X*  2x2 2x1

49. 49 Matrix Formulation/GLM Intercept: b Slope: m Age x1x2 y2 y1 y = X*  y1 y2 1 x1 1 x2 bmbm =* 1 X = x1 1x2 # of Cols of X = # Parameters Each parameter has a column Each column means something Parameter meaning from column

50. 50 GLM Solution Intercept: b Slope: m Age x1x2 y2 y1 y = X*   ultiply both sides by X -1  =X -1 *y y1 y2 1 x1 1 x2 bmbm =* 1 X = x1 1x2 X -1 = -x1 1 11  = x2-x1 Non-invertible if x1=x2 Ill-conditioned if x1 near x2 -- Sensitive to noise

51. 51 Non-invertibility Intercept: b Slope: m Age x1x2 y2 y1 y2 1 x1 1 x2 bmbm =* 1 X =.1 1 y = X*  1.2 y = 1.2 1  = 2.2  = 10 1.2 y = 1.2 x1=x2=0.1 Want to find  but …  is not unique!

52. 52 More than Two Points y1 = 1*b + x1*m y2 = 1*b + x2*m y3 = 1*b + x3*m Intercept: b Slope: m y = X*   ultiply both sides by X +  =X + y=(X’X) -1 X’y Three Eqns, Two Unknowns Over-determined Still two columns in X, same meaning X is not square (use pseudo-inv X + ; LMS) Same invertibility concerns Measure noise (DOF != 0)

53. Matrix Decomposition/Factorization Break up a single matrix into several matrices that are multiplied together: Eigen System Singular Value Decomposition (SVD) Principle Component Analysis (PCA) Cholesky Decomposition - “square root” of a matrix

54. Eigen Decomposition A = Q*  *Q -1, A*v = *v A – square matrix v – eigenvector - eigenvalue Q – matrix of eigenvectors  – diagonal matrix of eigenvalues