1. Multivariate Dyadic Regression Trees for Sparse Learning Problems Xi Chen Machine Learning Department Carnegie Mellon University (joint work with Han Liu)

2. Content Experimental Results Statistical Property Multivariate Regression and Dyadic Regression Tree Tree Learning Algorithm Multivariate Dyadic Regression Tree for Sparse Learning

3. Multivariate Regression Model Predictors Responses Estimate : Minimize the L 2 -risk Empirical Risk Minimization

4. Tree Based Method  Estimation using tree based methods  Why trees?  Simplicity of Design  Good Interpretability  Easy Implementation  Good Practical Performance

5. Tree Based Method  CART (Classification and Regression Tree) [Breiman 1984] No. of terminal nodes Hard to be theoretically analyzed!

6. Dyadic Decision/Regression Tree  Dyadic Split [Scott 2004]

7. Sparse Model Lower Minimax Rate of Convergence of the risk Slow Fast Sparse Model

8. Regression Tree Piecewise Constant Piecewise Linear Piecewise Polynomial Gamma-Ray Burst 845

9. Multivariate Dyadic Regression Tree (MDRT) Active Set Rule 1 Rule 2 Multivariate Dyadic Regression Tree (MDRT) Variable Selection

10. Multivariate Dyadic Regression Tree Regularization Parameter Fine partitionSparse Model Lower degree poly

11. Statistical Property  Assumption 1:  Assumption 2:  Convergence Rate Minimax Rate

12. Tree Learning Algorithm Loss: Minimize the cost

13. Tree Learning Algorithm  Tree-growing stage  Pruning-back stage Randomized Greedy

14. Experimental Results  Methods Compared Methods Greedy MDRT with M=1 MDRT(G, M=1) Randomized MDRT with M=1 MDRT(R, M=1) Greedy MDRT with M=0 MDRT(G, M=0) Randomized MDRT with M=0 MDRT(R, M=0) Classification and Regression Tree CART Piecewise Linear Piecewise Constant

15. Generalized Nonlinear Model Experimental Results  Synthetic Data Linear Model Additive Model

16. Experimental Results

17.  Real Data (MSE) 10 artificial variables from Unif(0,1) 15 artificial variables from Unif(0,1) Never selected in 20 runs for M=1

18. Conclusion  Multivariate Regression Tree Model  Dyadic Split  A novel penalization term  Theoretically, achieve nearly optimal minimax rate for (α,C) smooth function  Empirically, conduct variable selection for sparse models  Efficient computation tree learning algorithm  Extensions  Classification Trees  Forest Extensions