1. Markus Uhr Feature Extraction Sparse, Flexible and Efficient Modeling using L 1 -Regularization Saharon Rosset and Ji Zhu

2. Sparse Modeling Using L 1 -Regularization Feature Extraction Contents 1. Idea 2. Algorithm 3. Results

3. Sparse Modeling Using L 1 -Regularization Feature Extraction Part 1: Idea

4. Sparse Modeling Using L 1 -Regularization Feature Extraction Introduction Setting: Implicit dependency on training data Linear model (  u se  -functions) Model:

5. Sparse Modeling Using L 1 -Regularization Feature Extraction Introduction Problem: How to choose weight  of regularization? Answer: Find for all  [0,  ) Can this be done efficiently (time, memory)? Yes, if we impose restrictions on

6. Sparse Modeling Using L 1 -Regularization Feature Extraction Restrictions shall be piecewise linear What impact on L(w) and J(w)? Can we still solve real world problems?

7. Sparse Modeling Using L 1 -Regularization Feature Extraction Restrictions must be piecewise constant L(w) quadratic in w J(w) linear in w

8. Sparse Modeling Using L 1 -Regularization Feature Extraction Quadratic Loss Functions square loss in regression hinge loss for classification (  SVM)

9. Sparse Modeling Using L 1 -Regularization Feature Extraction Linear Penalty Functions Sparseness property

10. Sparse Modeling Using L 1 -Regularization Feature Extraction Bet on Sparseness 50 samples with 300 independent Gaussian variables 1.Row: 3 non-zero variables 2.Row: 30 non-zero variables 3.Row: 300 non-zero variables

11. Sparse Modeling Using L 1 -Regularization Feature Extraction Part 2: Algorithm

12. Sparse Modeling Using L 1 -Regularization Feature Extraction „Linear Toolbox“ a(r), b(r) and c(r) piecewise constant coefficients Regression Classification

13. Sparse Modeling Using L 1 -Regularization Feature Extraction Optimization Problem

14. Sparse Modeling Using L 1 -Regularization Feature Extraction Algorithm Initialization start at t=0  w=0 determine set of non-zero components starting direction

15. Sparse Modeling Using L 1 -Regularization Feature Extraction Algorithm Loop follow the direction until one of the following happens: addition of new component vanishing of a non-zero component hit of a “knot” (discontinuity of a(r), b(r), c(r) )

16. Sparse Modeling Using L 1 -Regularization Feature Extraction Algorithm Loop direction update stopping criterion

17. Sparse Modeling Using L 1 -Regularization Feature Extraction Part 3: Results

18. Sparse Modeling Using L 1 -Regularization Feature Extraction NIPS Results General procedure 1.pre-selection (univariate t-statistic) 2.Algorithm loss function: Huberized hinge loss 3.Find best * based on validation dataset

19. Sparse Modeling Using L 1 -Regularization Feature Extraction NIPS Results Dexter Dataset m=300, n=20'000, pre-selection: n=1152 linear pieces of : 452 Optimum at (  120 non-zero components)

20. Sparse Modeling Using L 1 -Regularization Feature Extraction NIPS Results Not very happy with the results  working with the original variables  simple linear model  L 1 regularization for feature selection

21. Sparse Modeling Using L 1 -Regularization Feature Extraction Conclusion theory  practice limited to linear classifier other extensions Regularization Path for the SVM (L 2 )