1. Informatics and Mathematical Modelling / Intelligent Signal Processing ISCAS 2008 1 Morten Mørup Approximate L0 constrained NMF/NTF Morten Mørup Informatics and Mathematical Modeling Technical University of Denmark Work done in collaboration with Professor Lars Kai Hansen Informatics and Mathematical Modeling Technical University of Denmark PhD Kristoffer Hougaard Madsen Informatics and Mathematical Modeling Technical University of Denmark

2. Informatics and Mathematical Modelling / Intelligent Signal Processing ISCAS 2008 2 Morten Mørup Non-negative Matrix Factorization (NMF) VWH, V≥0,W≥0, H≥0 NMF gives Part based representation! (Lee & Seung – Nature 1999) 

3. Informatics and Mathematical Modelling / Intelligent Signal Processing ISCAS 2008 3 Morten Mørup NMF based on Multiplicative updates Step size parameter

4. Informatics and Mathematical Modelling / Intelligent Signal Processing ISCAS 2008 4 Morten Mørup fast Non-Negative Least Squares, fNNLS Active Set procedure (Lawson and Hanson, 1974)

5. Informatics and Mathematical Modelling / Intelligent Signal Processing ISCAS 2008 5 Morten Mørup NMF not in general unique!! V=WH=(WP)(P -1 H)=W ’ H ’ (Donoho & Stodden, 2003)

6. Informatics and Mathematical Modelling / Intelligent Signal Processing ISCAS 2008 6 Morten Mørup FIX: Impose sparseness (Hoyer, 2001,2004 Eggert et al. 2004) Ensures uniqueness Eases interpretability (sparse representation  factor effects pertain to fewer dimensions) Can work as model selection (Sparseness can turn off excess factors by letting them become zero) Resolves over complete representations (when model has many more free variables than data points) L1 used as convex proxy for the L0 norm, i.e. card(H)

7. Informatics and Mathematical Modelling / Intelligent Signal Processing ISCAS 2008 7 Morten Mørup Least Angle Regression and Selection(LARS)/Homotopy Method  

8. Informatics and Mathematical Modelling / Intelligent Signal Processing ISCAS 2008 8 Morten Mørup Controlling sparsity degree (Patric Hoyer 2004) Controlling sparsity degree (Mørup et al., 2008) Sparsity can now be controlled by evaulating the full regularization path of the NLARS

9. Informatics and Mathematical Modelling / Intelligent Signal Processing ISCAS 2008 9 Morten Mørup New Algorithm for sparse NMF: 1: Solve for each column of H using NLARS and obtain solutions for all values of  (i.e. the entire regularization path) 2: Select -solution giving the desired degree of sparsity 3: Update W such that || W d || F =1, according to (Eggert et al. 2004) Repeat from step 1 until convergence

10. Informatics and Mathematical Modelling / Intelligent Signal Processing ISCAS 2008 10 Morten Mørup CBCL face database USPS handwritten digits

11. Informatics and Mathematical Modelling / Intelligent Signal Processing ISCAS 2008 11 Morten Mørup

12. Informatics and Mathematical Modelling / Intelligent Signal Processing ISCAS 2008 12 Morten Mørup Conclusion New efficient algorithm for sparse NMF based on the proposed non-negative version of the LARS algorithm The obtained full regularization path admit to use L1 as a convex proxy for the L0 norm to control the degree of sparsity given by The proposed method is more efficient than previous methods to control degree of sparsity. Furhtermore, NLARS is even comparable in speed to the classic efficient fNNLS method. Proposed method directly generalizes to tensor decompositions through models such as Tucker and PARAFAC when using an alternating least squares approach.