Non-negative Matrix Factorization Recent algorithms, extensions and available software Atina Dunlap Brooks (adbrook2@stat.ncsu.edu) North Carolina State University

Recent Algorithms Lee & Seung’s multiplicative updates are easy to understand and to implement Can be very slow to converge ALS can speed things up Convergence theory is not particularly strong Most NMF methods do not have robust convergence, but work well in practice

Projected Gradient Descent Method Chih-Jen Lin (2007) Bound-constrained optimization Projected Gradient

Projected Gradient Descent Method Can be applied to both the multiplicative updates and the ALS solution Generally, greatest speed was achieved with the projected gradient combined with ALS

Fast Non-Negative Matrix Approximation Kim, Sra & Dhillon (2007) Employs Newton-type methods to solve NMF Uses curvature information vs. gradient descent approach Provide an exact method (good accuracy, but still slow) and a very fast inexact method

References for Algorithm Comparisons Algorithms and Applications for Approximate Nonnegative Matrix Factorization by Berry, Browne, Langville, Pauca & Plemmons (2006) Optimality, Computation, and Interpretations of Nonnegative Matrix Factorizations by Chu, Diele, Plemmons & Ragni (2004)

Extensions Tri-Factorization Semi-NMF Convex-NMF Non-negative Tensor Factorization Inferential Robust Matrix Factorization

Orthogonal Tri-factorization Ding, Li, Peng & Park (2006) Requiring orthogonality introduces uniqueness and improves clustering interpretations A = WSH, where WTW=I and HTH=I W gives row clusters while H gives column clusters

Semi-NMF Ding, Li & Jordan (2006) Allows A and W to contain negative values, but H is restricted to non-negative Provides more flexibility (negative entries) and a clustering which is usually better than k-means

Convex-NMF Ding, Li & Jordan (2006) Restricts W to be convex combinations of the columns of A Ensures meaningful cluster centroids W and H tend to be sparse

Non-Negative Tensor Factorization Uses n-way arrays instead of the 2-dimensional arrays used by NMF Presentations during the workshop by Michael Berry and Bob Plemmons

Inferential Robust Matrix Factorization Fogel, Young, Hawkins & Ledirac (2007) Uses the same method for robustness as Liu et al. (2003) for robust SVD Paul Fogel will be presenting on an application

Software - Matlab Matlab Code Patrik Hoyer Chih-Jen Lin http://www.cs.helsinki.fi/u/phoyer/ Includes Lee & Seung’s multiplicative updates and Hoyer’s sparseness Chih-Jen Lin http://www.csie.ntu.edu.tw/~cjlin/nmf/ Includes projected gradient descent applied to multiplicative updates and ALS

Software C code – nnmf() JMP script - irMF Simon Sheperd Paul Fogel http://www.simonshepherd.supanet.com/nnmf.htm Very fast algorithm (as of 2004) JMP script - irMF Paul Fogel http://www.niss.org/irMF/ Inferential Robust Matrix Factorization

Thank You