1. Algorithm for non-negative matrix factorization Daniel D. Lee, H. Sebastian Seung. Algorithm for non-negative matrix factorization. Nature.

2. Unsupervised learning algorithms Unsupervised learning algorithms: –PCA –VQ (vector quantization) –NMF

3. NMF –Triangular Factorization (LU) cannot work. It should be a square matrix. –QR Factorization MATLAB use qr function to do QR Factorization, the syntax is –[Q,R]= qr(A) –Q is a orthonormal matrix QQ T =I –R is a upper triangle matrix –A is a m*n matrix –iterative m*rr*nr < n or m

4. Cost function A,B are two non-negative matrices Count the distance between A and B –Euclidean distance: –Kullback-Leibler distance (Relative Entropy)

5. Cost function These two equations are both lower bounded by zero. Proof

6. Goal Minimize with respect to and subject to the constraints, Gradient descent : –simplest technique to implement. –convergence can be slow. –inconvenient for large applications. Conjugate gradient : –Faster convergence, at least in the vicinity of local minima. –Complicated to implement.

7. Multiplicative update rules Euclidean distance Divergence

8. Multiplicative versus additive update rules

9. Definition 1: is an auxiliary function for F(h) if the conditions are satisfied. Lemma 1: proof: Note: only if is a local minimum of

10. Lemma 2 If is the diagonal matrix

11. Proof