Structured Sparse Principal Component Analysis Reading Group Presenter: Peng Zhang Cognitive Radio Institute Friday, October 01, 2010 Authors: Rodolphe Jenatton, Guillaume Obozinski, Francis Bach

Outline ■Introduction (in Imaging Sense) □Principal Component Analysis (PCA) □Sparse PCA (SPCA) □Structured Sparse PCA (SSPCA) ■Problem Statement ■The SSPCA Algorithm ■Experiments ■Conclusion and Other Thoughts

Introduction (Imaging Sense) ■The face recognition problem □A database includes a huge amount of faces □How to let computer to recognize different faces with database ■The challenge □Huge amount of data □Computation complexity ■The trick □Represent the face using a weighted “face dictionary” ▪Similar to code book in data compression ▪Example: An 200 X 200 pixel face can be represented by 100 coefficients using the “face dictionary” ■The solution □Principal component analysis (PCA)

PCA ■PCA □A compression method □Given a large amount of sample vectors {x} □2 nd moment statistics of the sample vectors □Eigen-decomposition finds the “dictionary” and “energy” of the dictionary codes ▪Eigen-vectors {v} form the “dictionary” ▪Eigen-values {d} give the “energy” of “dictionary” elements

PCA ■Original signal can be represented using only part of the dictionary ▪Data is compressed with fewer elements ■Meaning of “dictionary” v: □It is the weights of each elements in x ■The problem for PCA for face recognition: No physical meaning for “dictionary”

PCA ■Face recognition The Face Samples PCA The “dictionary”, eigen-faces These eigen-faces can reconstruct original faces perfectly, but make no sense in real life

Structured SPCA ■The SPCA goal: □Make dictionary more interpretable □The “sparse” solution: Limit the number of nonzeros Non-sparse Eigen-faces from PCA Sparse Eigen-faces from SPCA But the eigen-faces are still meaningless most of time

Structured SPCA ■The new idea, SSPCA □Eigen-faces will be meaningful when some structured constraints are set □Meaningful areas in faces are constrained in “grids” Eigen-faces from SSPCA

Structured SPCA ■This paper’s contribution □Add the “structure” constraint to make the dictionary more meaningful □How the constraint works □Meaningful dictionary is more close to “true” dictionary □Meaningful dictionary is more robust against noise □Meaningful dictionary is more accurate in face recognition

Outline ■Introduction □Principal Component Analysis (PCA) □Sparse PCA (SPCA) □Structured Sparse PCA (SSPCA) ■Problem Statement ■The SSPCA Algorithm ■Experiments ■Conclusion and Other Thoughts

Problem Statement ■From SPCA to SSPCA □The optimization problem □X is sample matrix, U is coefficient matrix, V is dictionary □||.|| and are different types of norms □The trick in SPCA ▪L1 norm force the dictionary to be a sparse solution

Problem Statement ■Structured SPCA, however, deal with a mixed l1/l2 minimization: ■Right now it’s hard for me to understand the G and d

Problem Statement ■In short, the norm constraints have the following effects □Dictionary has some structures □All non-zeros in the dictionary will be confined inside a grid

Outline ■Introduction □Principal Component Analysis (PCA) □Sparse PCA (SPCA) □Structured Sparse PCA (SSPCA) ■Problem Statement ■The SSPCA Algorithm ■Experiments ■Conclusion and Other Thoughts

The SSPCA Algorithm ■Making the dictionary sparser □The norm, □The new SSPCA problem:

The SSPCA Algorithm ■Methods to solve a sequence of convex problems

Excerpt from Author’s slide ■Excerpt from author’s slide:

Excerpt from Author’s slide

Outline ■Introduction □Principal Component Analysis (PCA) □Sparse PCA (SPCA) □Structured Sparse PCA (SSPCA) ■Problem Statement ■The SSPCA Algorithm ■Experiments ■Conclusion and Other Thoughts

Conclusion and Other Thoughts ■Conclusion □This paper shows how to use SSPCA □SSPCA gets better performance in denoising, face recognition and classification ■Other thoughts □Usually, the meaningful dictionary in communication signals is Fourier dictionary □But Fourier dictionary may not fit some transient signals or time- variant signals □How to manipulate the G, d and norms to set constraints for our needs?

THANK YOU!