Algorithm for non-negative matrix factorization Daniel D. Lee, H. Sebastian Seung. Algorithm for non-negative matrix factorization. Nature.

Unsupervised learning algorithms Unsupervised learning algorithms: –PCA –VQ (vector quantization) –NMF

NMF –Triangular Factorization (LU) cannot work. It should be a square matrix. –QR Factorization MATLAB use qr function to do QR Factorization, the syntax is –[Q,R]= qr(A) –Q is a orthonormal matrix QQ T =I –R is a upper triangle matrix –A is a m*n matrix –iterative m*rr*nr < n or m

Cost function A,B are two non-negative matrices Count the distance between A and B –Euclidean distance: –Kullback-Leibler distance (Relative Entropy)

Cost function These two equations are both lower bounded by zero. Proof

Goal Minimize with respect to and subject to the constraints, Gradient descent : –simplest technique to implement. –convergence can be slow. –inconvenient for large applications. Conjugate gradient : –Faster convergence, at least in the vicinity of local minima. –Complicated to implement.

Multiplicative update rules Euclidean distance Divergence

Multiplicative versus additive update rules

Definition 1: is an auxiliary function for F(h) if the conditions are satisfied. Lemma 1: proof: Note: only if is a local minimum of

Lemma 2 If is the diagonal matrix

Proof