1. Introduction to Matrices
Douglas N. Greve

2. Why Matrices? Simplifies notation Simplifies concepts
Grounding for general linear model (GLM)
Simplifies implementation (eg, matlab, octave)
Optimization

3. M M = M What is the matrix? 2 4 y p 3.3i -7.4
A matrix is a table of numbers, like a spread sheet
Each entry is called an “element”
There is a value in each element (no empty cells)
Elements can be whole numbers, positive, negative, fractions, real, imaginary, symbolic variables 2 4 y p
3.3i
-7.4 M
M = M

4. Parts of Matrices Size/Dimensions Elements Groupings of Elements
Diagonals
Triangles
Vectors

5. Matrix Size/Dimension
Number of Rows
Number of Columns
Rows-by-Columns (RowsxCols) 2 4 y p
3.3i
-7.4 2 4 y
1x3
2x3 2 p p
2x1
1x1

6. g = -1 g = [-1] Scalar versus Matrix Scalar is just a number or value
No dimension or size
g = -1
No brackets
No dimension
Not a 1x1
g = [-1]
1x1

7. M(row,col) M = M(i,j) Mij M(2,3) = -7.4 Matrix Element 2 4 y p 3.3i
Indicated by its row and column number
Scalar value
M(row,col) 2 4 y p
3.3i
-7.4
M =
M(i,j)
2x3
Mij
(1,1)
(1,2)
(1,3)
M(2,3) = -7.4
(2,1)
(2,2)
(2,3)

8. } } Off-Diagonals Diagonals 2 4 6 1 3 5 7 8 9 2 4 6 1 3 5 7 8 9
Row = Column + Offset 7 8 9
3x3 2 4 6 1 3 5
Row = Column - 2 (Second Upper Diagonal) } 7 8 9
Row = Column - 1 (First Upper Diagonal)
Row = Column + 0 (Main Diagonal)
Row = Column - 1 (First Lower Diagonal)
} Off-Diagonals
Row = Column - 2 (Second Lower Diagonal)

9. Triangles 2 4 6 1 3 5 7 8 9
Upper Triangle
3x3
Lower Triangle

10. Vector 2 4 y 2 p Special Type of Matrix Number of Rows or Columns = 1
Often given lower-case variable names 2 4 y
1x3 Row Vector 2 p
2x1 Column Vector

11. A matrix consists of vectors2 4 y p
3.3i
-7.4
2x3 2 4 x p
3.3i
-7.4 2 4 x p
3.3i
-7.4
Two 1x3 row vectors
Three 2x1 column vectors

12. Special Matrices Square Diagonal Identity Triangular Symmetric
Toeplitz

13. Square Matrix
Rows = Columns 2 4 p
1x1 p
3.3i
2x2

14. Diagonal Matrix 2 3 9 All off-diagonal elements = 0 M(i,j)=0 if i != j3 9
3x3

15. I = Identity Matrix 1 1 1 Main diagonal has all 1s
All off-diagonal elements = 0
Square
Symbol “I” 1
I = 1 1
3x3
Any matrix multiplied by the identity is itself

16. Triangular Matrices 2 4 6 3 5 9 2 1 3 7 8 9 3x3 3x33 5 9
3x3 2 1 3 7 8 9
3x3
Upper Triangular – all values below the main diagonal are 0
Lower Triangular – all values above the main diagonal are 0

17. Symmetric Matrix 2 4 6 4 3 5 6 5 9 M(row,col) = M(col,row)
Reflect across main diagonal
Square 2 4 6 4 3 5 6 5 9
3x3

18. Toeplitz Matrix
All elements on a diagonal are the same 2 4 6 2 4 1 2 4 1 2 7 1 2
3x3 7 1
3x2
A causal, time invariant linear system is represented by an upper triangular Toeplitz matrix

19. Ones Matrix 1 1 1 1 1 1 1 1 1 1 1 1 All elements are 1
Any number of rows or columns 1 1 1 1 1 1
1x3 1 1 1
2x3 1 1 1
2x1
1x1

20. Zeros Matrix All elements are 0 Any number of rows or columns 1x3 2x3
1x3
2x3
2x1
1x1

21. Matrix Operations Addition/Subtraction Vector Multiplication
Matrix Multiplication
Multiplication by scalar
Transpose
Trace
Inverse
Pseudo Inverse

22. Matrix Addition/Subtraction
Element-by-Element Addition/Subtraction
Size of two matrices must be the same
C(row,col) = A(row,col) +/- B(row,col)
C = A + B (your first matrix equation!) 2 4 6 7 9
A =
B = 1 3 5 1 -2
3.3
2x3
2x3 2 11 15
C = 2 1
8.3
2x3

23. Vector Multiplication
One Row Vector and One Column Vector
Same length, Results in 1x1
Inner Product, Dot Product, “Scalar” Product
Like a correlation
“Multiply and accumulate”
z = x.y= x*y = S xi*yi 2 4 6
x =
1x3
* * *
= 46 7 1
y =
3x1

24. Matrix Multiplication
Series of vector multiplications
Same “inner dimension” 4 2 8 2 4 6
A =
B = 1 5 6 2 1 3 5 3 7 9 3
2x3
3x4
c11
c12
c13
c14
cij=Arowi*Bcolj
C=A*B =
c21
c22
c23
c24
2x4
Dimensions: (2x3) *(3x4)=(2x4)

25. Matrix Multiplication
Series of vector multiplications
Same “inner dimension” 4 2 8 2 4 6
A =
B = 1 5 6 2 1 3 5 3 7 9 3
2x3
3x4
c11
c12
c13
c14
c11= 2 4 6 1 3
C=A*B =
c21
c22
c23
c24
2x4
2*0+4*1+6*3= 22

26. Matrix Multiplication
Series of vector multiplications
Same “inner dimension” 4 2 8 2 4 6
A =
B = 1 5 6 2 1 3 5 3 7 9 3
2x3
3x4 22
c12
c13
c14
c12= 2 4 6 4 5 7
C=A*B =
c21
c22
c23
c24
2x4
2*4+4*5+6*7= 70

27. Matrix Multiplication
Series of vector multiplications
Same “inner dimension” 4 2 8 2 4 6
A =
B = 1 5 6 2 1 3 5 3 7 9 3
2x3
3x4 22 70
c13
c14
c13= 2 4 6 2 6 9
C=A*B =
c21
c22
c23
c24
2x4
2*2+4*6+6*9= 82

28. Matrix Multiplication
Series of vector multiplications
Same “inner dimension” 4 2 8 2 4 6
A =
B = 1 5 6 2 1 3 5 3 7 9 3
2x3
3x4 22 70 82
c14
c14= 2 4 6 8 2 3
C=A*B =
c21
c22
c23
c24
2x4
2*8+4*2+6*3= 42

29. Matrix Multiplication
Series of vector multiplications
Same “inner dimension” 4 2 8 2 4 6
A =
B = 1 5 6 2 1 3 5 3 7 9 3
2x3
3x4 22 70 82 42
c21= 1 3 5 1 3
C=A*B =
c21
c22
c23
c24
2x4
1*0+4*1+5*3= 18

30. Matrix Multiplication
Series of vector multiplications
Same “inner dimension” 4 2 8 2 4 6
A =
B = 1 5 6 2 1 3 5 3 7 9 3
2x3
3x4 22 70 82 42
c22= 1 3 5 4 5 7
C=A*B = 18
c22
c23
c24
2x4
1*4+4*5+5*7= 54

31. Outer Product (Vector Multiplication)
One Row Vector one Column Vector
Same length (N)
Result is NxN matrix
Special case of Matrix Multiplication
z = y*x 2 4 6 2 4 6
x =
1x3
1x3 7 1 7 1
y = 2 4 6
3x1 14 28 42
3x3
3x1

32. g = -1 gA = Multiplication with a scalar 2 4 6 1 3 5 -2 -4 -6 -1 -3 -5
Multiply each element by scalar value 2 4 6
g = -1
A =
No brackets
No dimension
Not a 1x1 1 3 5
2x3
gA = -2 -4 -6 -1 -3 -5
2x3

33. Matrix Transpose 2 1 2 4 6 4 3 1 3 5 6 5 “Reflect” across diagonal
B(row,col) = A(col,row) 2 1 2 4 6 4 3
A =
B=AT =At =A’= 1 3 5 6 5
2x3
3x2
If A=A’, then A is symmetric (must be square)

34. Trace 2 4 6 1 3 5 7 8 9 Sum of diagonal Elements Trace = S M(i,i) M =

35. D = c11* c22 - c12* c21 Determinant c11 c12 c21 c22
Property of a square matrix
Complicated in general
Simple for a 2x2
c11
c12
D = c11* c22 - c12* c21
C =
c21
c22
2x2

36. 1 D D = c11* c22 - c12* c21 Matrix Inverse c11 c12 c22 -c12 c21 c22
C*A = I, then A=C-1 and C=A-1
Must be square
Complicated in general
Simple for a 2x2
c11
c12
c22
-c12 1 D
C =
C-1 =
c21
c22
-c21
c11
2x2
2x2
D = c11* c22 - c12* c21

37. 1 Invertibility D = 1.0*1.0 - 2.0*0.5 = 1-1 = 0 IMPORTANT!!!
Not all matrices are invertible
D=0
“Singular”
D = 1.0* *0.5 = 1-1 = 0
1.0
2x2
C =
2.0
0.5
1.0
2x2
C-1 =
-2.0
-0.5 1

38. D = 1.0*1.0 - 2.0*0.5 = 1-1 = 0 Singularity and “Ill-Conditioned”
2x2
Column 2 = twice Column 1
Linear Dependence
Ill-Conditioned: D is “close” to 0
Relates to efficiency of a GLM.
D = 1.0* *0.5 = 1-1 = 0

39. If X has more columns than rows X+ = (X’*X)-1*X’
Pseudo-Inverse
Can be used for non-square
Same rules on invertibility apply
Least-mean-square (LMS) for over-determined system
If X has more columns than rows
X+ = (X’*X)-1*X’
Check: X+*X = (X’*X)-1*X’*X=I
If X has more rows than columns
X+ = X’ *(X*X’)-1*
Check: X*X+ = X*X’ *(X*X’)-1=I

40. Matrix Applications

41. length(y) = x’*x = 1*1 + 1*1 + 1*1 = 3
Sum/Length/Average of a List
y =
3x1 1 2 3
x =
3x1 1
x’ = [1 1 1]1x3
“Ones vector”
sum(y) = x’*y = 1*1 + 1*2 + 1*3 = 6
1x3
3x1
length(y) = x’*x = 1*1 + 1*1 + 1*1 = 3
Average m = (x’*x)-1(x’*y) [Note: GLM]
= (3-1)*6 = 6/3 = 2

42. length(y) = x’*x = 0*0+ 1*1 + 1*1 = 2
Selective Average
y =
3x1 1 2 3
x =
3x1 1
x’ = [0 1 1]1x3
“Ones vector”
sum(y) = x’*y = 0*1 + 1*2 + 1*3 = 5
1x3
3x1
length(y) = x’*x = 0*0+ 1*1 + 1*1 = 2
Average m = (x’*x)-1(x’*y)
= (2-1)*5 = 5/2 = 2.5

43. y = y’ = [1 2 3]1x3 SS =S(y2) SS(y) = y’*y = 1*1 + 2*2 + 3*3 = 14
Sum of the Squares of a List
y =
3x1 1 2 3
y’ = [1 2 3]1x3
SS =S(y2)
SS(y) = y’*y = 1*1 + 2*2 + 3*3 = 14

44. DOF = x’*x - 1 = #RowsX - #ColsX
Variance of a List
y =
3x1 1 2 3
x =
3x1 1
s2 =S(yi-m)2
(N-1)
Residual e = y – m*x
s2 = sqrt(e’*e)/DOF
DOF = x’*x - 1 = #RowsX - #ColsX

45. Orthogonality z = x*y = S xi*yi 1 -1 5 1 * 2 + -1 * 3 + 5 * .2 = [0] 2
Vector product is 0
Property of two vectors
Uncorrelated/Independence - statistical property
z = x*y = S xi*yi 1 -1 5
x =
1x3
1 * 2 +
-1 * 3 +
5 * .2 = [0] 2 .2 3
y =
1x1
3x1

46. Fitting a Line Thickness x1 x2 y2 y1 Dependent Variable, Measurement
Subject 1
Subject 2
HRF Amplitude
IQ, Height, Weight
Age
Independent Variable

47. Linear Model System of Linear Equations y1 = 1*b + x1*m
Intercept: b
Slope: m
Age x1 x2 y2 y1
System of Linear Equations
y1 = 1*b + x1*m
y2 = 1*b + x2*m
Two Eqns, Two Unknowns
Solve 1st Eq for b
Substitute into 2nd Eq
Solve 2nd equation for m
Substitute m back into Eq for b
“Intercept” = “Offset”
m= (y2-y1)
(x2-x1)
b= (x2*y1-x1*y2)
(x2-x1)

48. Matrix Formulation/GLM
Intercept: b
Slope: m
Age x1 x2 y2 y1
y1 = 1*b + x1*m
y2 = 1*b + x2*m y1 y2
1 x1
1 x2 b m = *
2x1
2x1
2x2
y = X*b
- parameter vector
X - design matrix

49. Matrix Formulation/GLM
Intercept: b
Slope: m
Age x1 x2 y2 y1 y1 y2
1 x1
1 x2 b m = *
y = X*b
# of Cols of X = # Parameters
Each parameter has a column
Each column means something
Parameter meaning from column 1
X = x1 x2

50. GLM Solution y = X*b b=X-1*y = x2-x1 Multiply both sides by X-1
Intercept: b
Slope: m
Age x1 x2 y2 y1
y = X*b
Multiply both sides by X-1
b=X-1*y 1
X = x1 x2
X-1 =
-x1 -1 D
= x2-x1
Non-invertible if x1=x2
Ill-conditioned if x1 near x2
-- Sensitive to noise y1 y2
1 x1
1 x2 b m = *

51. Non-invertibility x1=x2=0.1 Want to find b, but … y = X*bm = *
y = X*b 1
b = 2
1.2
y = 1
X = .1
1.2
y =
Intercept: b
Slope: m
Age x1 x2 y2 y1 .2
b = 10
b is not unique!

52. More than Two Points y = X*b b=X+y=(X’X)-1X’y y1 = 1*b + x1*m
Intercept: b
Slope: m
y1 = 1*b + x1*m
y2 = 1*b + x2*m
y3 = 1*b + x3*m
y = X*b
Multiply both sides by X+
b=X+y=(X’X)-1X’y
Three Eqns, Two Unknowns
Over-determined
Still two columns in X, same meaning
X is not square (use pseudo-inv X+; LMS)
Same invertibility concerns
Measure noise (DOF != 0)

53. Matrix Decomposition/Factorization
Break up a single matrix into several matrices that are multiplied together:
Eigen System
Singular Value Decomposition (SVD)
= Principle Component Analysis (PCA)
Cholesky Decomposition - “square root” of a matrix
Independent Component Analysis (ICA)

54. Q – matrix of eigenvectors L – diagonal matrix of eigenvalues
Eigen Decomposition
A = Q*L*Q-1, A*v = l*v
A – square matrix
v – eigenvector
- eigenvalue
Q – matrix of eigenvectors
L – diagonal matrix of eigenvalues