1. Classification Derek Hoiem CS 598, Spring 2009 Jan 27, 2009

2. Outline Principles of generalization Survey of classifiers Project discussion Discussion of Rosch

3. Pipeline for Prediction ImageryRepresentationClassifierPredictions

4. Free Lunch Theorem

5. Bias and Variance Complexity Low Bias High Variance High Bias Low Variance Error

6. Overfitting Need validation set Validation set not same as test set

7. Bias-Variance View of Features More compact = lower variance, potentially higher bias More features = higher variance, lower bias More independence among features = simpler classifier  lower variance

8. How to reduce variance Parameterize model E.g., linear vs. piecewise

9. How to measure complexity? VC dimension Training error + Upper bound on generalization error N: size of training set h: VC dimension  : 1-probability

10. How to reduce variance Parameterize model Regularize

11. How to reduce variance Parameterize model Regularize Increase number of training examples

12. Effect of Training Size Number of Training Examples Error

13. Risk Minimization Margins xx x x x x x x o o o o o x2 x1

14. Classifiers Generative methods – Naïve Bayes – Bayesian Networks Discriminative methods – Logistic Regression – Linear SVM – Kernelized SVM Ensemble methods – Randomized Forests – Boosted Decision Trees Instance based – K-nearest neighbor Unsupervised – Kmeans

15. Components of classification methods Objective function Parameterization Regularization Training Inference

16. Classifiers: Naïve Bayes Objective Parameterization Regularization Training Inference x1x1 x2x2 x3x3 y

17. Classifiers: Logistic Regression Objective Parameterization Regularization Training Inference

18. Classifiers: Linear SVM Objective Parameterization Regularization Training Inference xx x x x x x x o o o o o x2 x1

19. Classifiers: Linear SVM Objective Parameterization Regularization Training Inference xx x x x x x x o o o o o x2 x1

20. Classifiers: Linear SVM Objective Parameterization Regularization Training Inference xx x x x x x x o o o o o o x2 x1 Needs slack

21. Classifiers: Kernelized SVM Objective Parameterization Regularization Training Inference xxxxooo x1x1 x x x x o o o x1x1 x12x12

22. Classifiers: Decision Trees Objective Parameterization Regularization Training Inference xx x x x x x x o o o o o o x2 x1

23. Ensemble Methods: Boosting figure from Friedman et al. 2000

24. Boosted Decision Trees … Gray? High in Image? Many Long Lines? Yes No Yes Very High Vanishing Point? High in Image? Smooth?Green? Blue? Yes No Yes Ground Vertical Sky [Collins et al. 2002] P(label | good segment, data)

25. Boosted Decision Trees How to control bias/variance trade-off – Size of trees – Number of trees

26. K-nearest neighbor xx x x x x x x o o o o o o o x2 x1 Objective Parameterization Regularization Training Inference

27. Clustering xx x x x x o o o o o x1 x x2 ++ + + + + + + + + + x1 +

28. References General – Tom Mitchell, Machine Learning, McGraw Hill, 1997 – Christopher Bishop, Neural Networks for Pattern Recognition, Oxford University Press, 1995 Adaboost – Friedman, Hastie, and Tibshirani, “Additive logistic regression: a statistical view of boosting”, Annals of Statistics, 2000 SVMs – http://www.support-vector.net/icml-tutorial.pdf

29. Project Idea Investigate various classification methods on several standard vision problems – At least five problems with pre-defined feature set and training/test set – Effect of training size – Effect of number of variables – Any method dominant? – Any guidelines for choosing method?

30. Project ideas?

31. Discussion of Rosch