+ Review of Linear Algebra Optimization 1/14/10 Recitation Sivaraman Balakrishnan

+ Outline Matrix subspaces Linear independence and bases Gaussian elimination Eigen values and Eigen vectors Definiteness Matlab essentials Geoff’s LP sketcher linprog Debugging and using documentation

Basic concepts Vector in R n is an ordered set of n real numbers. e.g. v = (1,6,3,4) is in R 4 A column vector: A row vector: m-by-n matrix is an object with m rows and n columns, each entry filled with a real (typically) number:

Basic concepts - II Vector dot product: Matrix product:

+ Matrix subspaces What is a matrix? Geometric notion – a matrix is an object that “transforms” a vector from its row space to its column space Vector space – set of vectors closed under scalar multiplication and addition Subspace – subset of a vector space also closed under these operations Always contains the zero vector (trivial subspace)

+ Row space of a matrix Vector space spanned by rows of matrix Span – set of all linear combinations of a set of vectors This isn’t always R n – example !! Dimension of the row space – number of linearly independent rows (rank) We’ll discuss how to calculate the rank in a couple of slides

+ Null space, column space Null space – it is the orthogonal compliment of the row space Every vector in this space is a solution to the equation Ax = 0 Rank – nullity theorem Column space Compliment of rank-nullity

+ Linear independence A set of vectors is linearly independent if none of them can be written as a linear combination of the others Given a vector space, we can find a set of linearly independent vectors that spans this space The cardinality of this set is the dimension of the vector space

+ Gaussian elimination Finding rank and row echelon form Applications Solving a linear system of equations (we saw this in class) Finding inverse of a matrix

+ Basis of a vector space What is a basis? A basis is a maximal set of linearly independent vectors and a minimal set of spanning vectors of a vector space Orthonormal basis Two vectors are orthonormal if their dot product is 0, and each vector has length 1 An orthonormal basis consists of orthonormal vectors. What is special about orthonormal bases? Projection is easy Very useful length property Universal (Gram Schmidt) given any basis can find an orthonormal basis that has the same span

+ Matrices as constraints Geoff introduced writing an LP with a constraint matrix We know how to write any LP in standard form Why not just solve it to find “opt”?

A special basis for square matrices The eigenvectors of a matrix are unit vectors that satisfy Ax = λ x Example calculation on next slide Eigenvectors are orthonormal and eigenvalues are real for symmetric matrices This is the most useful orthonormal basis with many interesting properties Optimal matrix approximation (PCA/SVD) Other famous ones are the Fourier basis and wavelet basis

Eigenvalues (A – λ I)x = 0 λ is an eigenvalue iff det(A – λ I) = 0 Example:

+ Projections (vector) (0,0,1) (0,1,0) (1,0,0) (2,2,2) a = (1,0) b = (2,2)

+ Matrix projection Generalize formula from the previous slide Projected vector = (Q T Q) -1 Q T v Special case of orthonormal matrix Projected vector = Q T v You’ve probably seen something very similar in least squares regression

Definiteness Characterization based on eigen values Positive definite matrices are a special sub-class of invertible matrices One way to test for positive definiteness is by showing x T Ax > 0 for all x A very useful example that you’ll see a lot in this class Covariance matrix

Matlab Tutorial - 1 Linsolve Stability and condition number Geoff’s sketching code – might be very useful for HW1

Matlab Tutorial - 2 Linprog – Also, very useful for HW1 Also, covered debugging basics and using Matlab help

+ Extra stuff Vector and matrix norms Matrix norms - operator norm, Frobenius norm Vector norms - L p norms Determinants SVD/PCA