1. LECTURE 09: CLASSIFICATION WRAP-UP February 24, 2016 SDS 293 Machine Learning

2. Announcement For those who have asked about extra credit for finishing the superhero LDA from Monday - Full data on the “training heroes” is available at: http://www.science.smith.edu/~jcrouser/SDS293/data/ - You may use R to calculate the covariance matrix if you like, but the rest must be done by hand - Anyone who submits a well-reasoned solution will have their lowest homework grade dropped Submissions should be turned in before spring break

3. Outline Finish up Monday’s pseudo-lab: classification approaches - 2-4 person teams (3 is ideal) - Take ~30 minutes to discuss the 6 scenarios, then we’ll come back and wrap up classification Evaluating models using resampling - Today: Cross-validation - Monday: Bootstrap

4. 30 min: comparing classification methods Work in teams of 2-4 people (3 is ideal) Goal: to deepen our understanding of how the 4 classification methods we’ve discussed measure up - K-nearest neighbors - Logistic regression - LDA - QDA Instructions: http://www.science.smith.edu/~jcrouser/SDS293/labs/lab6/ Can check your intuition against p.153-4 of ISLR

5. Discussion: classification methods Logistic Regression K-nearest neighbors LDAQDA Scenario 1 Scenario 2 Scenario 3 Scenario 4 Scenario 5 Scenario 6 ( )

6. Rough guide to choosing a method Is the decision boundary ~linear? Are the observations normally distributed? Do you have limited training data? Do you have limited training data? Start with LDA Start with LDA Start with Logistic Regression Start with Logistic Regression Start with QDA Start with QDA Start with K-Nearest Neighbors Start with K-Nearest Neighbors YES NO YES NO YES NO Linear methods Non-linear methods

7. Test vs. training In all of these methods, we evaluate performance using test error rather than training error (why?) Real life: if the data comes in stages (i.e. trying to predict election results), we have a natural “test set” Question: when that’s not the case, what do we do? Answer: set aside a “validation set”

8. What to set aside? Smarket: used years 2001-2004 to train, tested on 2005 Caravan: used the first 1000 records as test Carseats: - some of you split the data in half - some sampled randomly - some split on one of the predictors (population>300, sales>7, etc.) Question: did it make a difference?

9. Answer: absolutely Example: Auto dataset (392 observations) Goal: use linear regression to predict mpg using polynomial f (horsepower) Randomly split into 2 sets of 196 obs. (test and training) ^ 10x? Huge variation in prediction accuracy!

10. Issues with the “validation set” approach 1. The test error rate depends heavily on which observations are used for training vs. testing 2. We’re only training on a subset of the data (why is this a problem?) We need a new approach...

11. Coming up Monday: cross-validation A1/A2 feedback should be ready shortly A3 posted, due Feb. 29 th by 11:59pm