1. Lectures 15,16 – Additive Models, Trees, and Related Methods Rice ECE697 Farinaz Koushanfar Fall 2006

2. Summary Generalized Additive Models Tree-Based Methods PRIM – Bump Hunting Mutlivariate Adaptive Regression Splines (MARS) Missing Data

3. Additive Models In real life, effects are nonlinear Note: Some slides are borrowed from Tibshirani

4. Examples

5. The Price for Additivity Data from a study of Diabetic children, Predicting log C-peptide (a blood measurement)

6. Generalized Additive Models (GAM) Two-class Logistic Regression

7. Other Examples

8. Fitting Additive Models Given observations xi,yi, a criterion like the penalized sum of squares can be specified for this problem, where ’s are tuning parameters The mean of error term is zero!

9. Fitting Additive Models

10. The Backfitting Algorithm for Additive Models Initialize: Cycle: j=1,2,…,p,1,2,…,p,1,… Until the functions f j change less than a prespecified threshold

11. Fitting Additive Models (Cont’d)

12. Example: Penalized Least square

13. Example: Fitting GAM for Logistic Regression (Newton-Raphson Algorithm)

14. Example: Predicting Email Spam Data from 4601 mail messages, spam=1, email=0, filter trained for each user separately Goal: predict whether an email is spam (junk mail) or good Input features: relative frequencies in a message of 57 of the commonly occurring words and punctuation marks in all training set Not all errors are equal; we want to avoid filtering out good email, while letting spam get through is not desirable but less serious in its consequences

15. Predictors

16. Details

17. Some Important Features

18. Results Test data confusion matrix for the additive logistic regression model fit to the spam training data The overall test error rate is 5.3%

19. Summary of Additive Logistic Fit Significant predictors from the additive model fit to the spam training data. The coefficients represent the linear part of f ^ j, along with their standard errors and Z-score. The nonlinear p-value represents a test of nonlinearity of f ^ j

20. Example: Plots for Spam Analysis Figure 9.1. Spam analysis: estimated functions for significant predictors. The rug plot along the bottom of each frame indicates the observed values of the corresponding predictor. For many predictors, the nonlinearity picks up the discontinuity at zero.

21. In Summary Additive models are a useful extension to linear models, making them more flexible The backfitting procedure is simple and modular Limitations for large data mining applications Backfitting fits all predictors, which is not desirable when a large number are available

22. Tree-Based Methods

23. Node Impurity Measures

24. Results for Spam Example

25. Pruned tree for the Spam Example

26. Classification Rules Fit to the Spam Data

27. PRIM-Bump Hunting

28. Number of Observations in a Box

29. Basis Functions

30. MARS Forward Modeling Procedure

31. Multiplication of Basis Functions

32. MARS on Spam Example