1. CS479/679 Pattern Recognition Dr. George Bebis
Linear Algebra Review
CS479/679 Pattern Recognition Dr. George Bebis
George Bebis

2. n-dimensional Vector An n-dimensional vector v is denoted as follows:
The transpose vT is denoted as follows:

3. Inner (or dot) product
Given vT = (x1, x2, , xn) and wT = (y1, y2, , yn), their dot product defined as follows:
(scalar) or

4. Orthogonal / Orthonormal vectors
A set of vectors x1, x2, , xn is orthogonal if
A set of vectors x1, x2, , xn is orthonormal if k

5. Linear combinations
A vector v is a linear combination of the vectors v1, ..., vk if:
where c1, ..., ck are constants.
Example: vectors in R3 can be expressed as a linear combinations of unit vectors i = (1, 0, 0), j = (0, 1, 0), and k = (0, 0, 1)

6. Space spanning
A set of vectors S=(v1, v2, , vk ) span some space W if every vector in W can be written as a linear combination of the vectors in S
- The unit vectors i, j, and k span R3 w

7. Linear dependence
A set of vectors v1, ..., vk are linearly dependent if at least one of them is a linear combination of the others.
(i.e., vj does not appear on the right side)

8. Linear independence
A set of vectors v1, ..., vk is linearly independent if no vector can be represented as a linear combination of the remaining vectors, i.e.:
Example:
c1=c2=0

9. Vector basis
A set of vectors (v1, ..., vk) forms a basis in some vector space W if:
(1) (v1, ..., vk) are linearly independent
(2) (v1, ..., vk) span W
Standard bases:
R2 R3 Rn

10. Matrix Operations Matrix addition/subtraction Matrix multiplication
Add/Subtract corresponding elements.
Matrices must be of same size.
Matrix multiplication
m x p
m x n
q x p n
Condition: n = q

11. Identity Matrix

12. Matrix Transpose

13. Symmetric Matrices
Example:

14. Determinants 2 x 2 3 x 3 n x n Properties: (expanded along 1st column)
(expanded along kth column)
Properties:

15. Matrix Inverse
The inverse of a matrix A, denoted as A-1, has the property:
AA-1=A-1A=I
A-1 exists only if
Terminology
Singular matrix: A-1 does not exist
Ill-conditioned matrix: A is “close” to being singular

16. Matrix Inverse (cont’d)
Properties of the inverse:

17. Matrix trace
Properties:

18. Rank of matrix Example:
Equal to the dimension of the largest square sub-matrix of A that has a non-zero determinant.
Example:
has rank 3

19. Rank of matrix (cont’d)
Alternative definition: the maximum number of linearly independent columns (or rows) of A.
Example:
i.e., rank is not 4!

20. Rank of matrix (cont’d)

21. Eigenvalues and Eigenvectors
The vector v is an eigenvector of matrix A and λ is an eigenvalue of A if:
Geometric interpretation: the linear transformation implied by A can not change the direction of the eigenvectors v, only their magnitude.
(assume non-zero v)

22. Computing λ and v
To find the eigenvalues λ of a matrix A, find the roots of the characteristic polynomial:
Example:

23. Properties of λ and v
Eigenvalues and eigenvectors are only defined for square matrices.
Eigenvectors are not unique (e.g., if v is an eigenvector, so is kv).
Suppose λ1, λ2, ..., λn are the eigenvalues of A, then:

24. Matrix diagonalization
Given an n x n matrix A, find P such that:
P-1AP=Λ where Λ is diagonal
Solution: Set P = [v1 v vn], where v1,v2 ,. . . vn are the eigenvectors of A:

25. Matrix diagonalization (cont’d)
Example:

26. Matrix diagonalization (cont’d)
If A is diagonalizable, then the corresponding eigenvectors v1,v2 ,. . . vn form a basis in Rn

27. Are all n x n matrices diagonalizable P-1AP ?
An n x n matrix A is diagonalizable iff it has n linearly independent eigenvectors.
i.e., if P-1 exists, that is, rank(P)=n
Theorem: If the eigenvalues of A are all distinct, their corresponding eigenvectors are linearly independent (i.e., A is diagonalizable).

28. Are all n x n matrices diagonalizable P-1AP ? (cont’d)
λ1=λ2=1 and λ3=2
λ1=λ2=0 and λ3=-2
non-diagonalizable
diagonalizable

29. Matrix decomposition
If A is diagonalizable, then A can be decomposed as follows:

30. Special case: symmetric matrices
The eigenvalues of a symmetric matrix are real and its eigenvectors are orthogonal.
P-1=PT
A=PDPT=