1. Derek Hoiem CS 598, Spring 2009 Jan 27, 2009
Classification
Derek Hoiem
CS 598, Spring 2009
Jan 27, 2009

2. Outline Principles of generalization Survey of classifiers
Project discussion
Discussion of Rosch

3. Pipeline for Prediction
Imagery
Representation
Classifier
Predictions

4. Free Lunch Theorem

5. Bias and Variance Error Complexity Low Bias High Variance High Bias
Low Variance
Error

6. Overfitting
Need validation set
Validation set not same as test set

7. Bias-Variance View of Features
More compact = lower variance, potentially higher bias
More features = higher variance, lower bias
More independence among features = simpler classifier  lower variance

8. How to reduce variance
Parameterize model
E.g., linear vs. piecewise

9. How to measure complexity?
VC dimension
Upper bound on generalization error
Training error +
N: size of training set
h: VC dimension
: 1-probability

10. How to reduce variance
Parameterize model
Regularize

11. How to reduce variance Parameterize model Regularize
Increase number of training examples

12. Effect of Training Size
Number of Training Examples
Error

13. Risk Minimization
Margins x o x2 x1

14. Classifiers Generative methods Discriminative methods Ensemble methods
Naïve Bayes
Bayesian Networks
Discriminative methods
Logistic Regression
Linear SVM
Kernelized SVM
Ensemble methods
Randomized Forests
Boosted Decision Trees
Instance based
K-nearest neighbor
Unsupervised
Kmeans

15. Components of classification methods
Objective function
Parameterization
Regularization
Training
Inference

16. Classifiers: Naïve Bayes
Objective
Parameterization
Regularization
Training
Inference y x1 x2 x3

17. Classifiers: Logistic Regression
Objective
Parameterization
Regularization
Training
Inference

18. Classifiers: Linear SVM
Objective
Parameterization
Regularization
Training
Inference x o x2 x1

19. Classifiers: Linear SVM
Objective
Parameterization
Regularization
Training
Inference x o x2 x1

20. Classifiers: Linear SVM
Objective
Parameterization
Regularization
Training
Inference x o x2 x1
Needs slack

21. Classifiers: Kernelized SVM
Objective
Parameterization
Regularization
Training
Inference x o x1 x o x1
x12

22. Classifiers: Decision Trees
Objective
Parameterization
Regularization
Training
Inference x o x2 x1

23. Ensemble Methods: Boosting
figure from Friedman et al. 2000

24. Boosted Decision Trees
High in
Image?
Gray?
Yes No
Yes No
Smooth?
Green?
High in
Image?
Many Long
Lines? …
Yes
Yes No
Yes No
Yes No No
Blue?
Very High
Vanishing
Point?
Yes No
Yes No
P(label | good segment, data)
Ground Vertical Sky
[Collins et al. 2002]

25. Boosted Decision Trees
How to control bias/variance trade-off
Size of trees
Number of trees

26. K-nearest neighbor Objective Parameterization Regularization Training
Inference x o x2 x1

27. Clustering x2 + x1 x o x1

28. References SVMs General Adaboost
Tom Mitchell, Machine Learning, McGraw Hill, 1997
Christopher Bishop, Neural Networks for Pattern Recognition, Oxford University Press, 1995
Adaboost
Friedman, Hastie, and Tibshirani, “Additive logistic regression: a statistical view of boosting”, Annals of Statistics, 2000
SVMs

29. Project ideas?

30. Discussion of Rosch