1. CSE 473/573 Computer Vision and Image Processing (CVIP) Ifeoma Nwogu Lecture 24 – Classifiers 1

2. Schedule Last class – We continued on segmentation, specifically mean field and graph cuts but could not finish  Today – Classifiers Readings for today: – Forsyth and Ponce chapter 16 2

3. Classifiers Given a feature representation for images, how do we learn a model for distinguishing features from different classes? Take a measurement x, predict a bit (yes/no; 1/-1; 1/0; etc) Today: Nearest neighbor classifiers Linear classifiers: support vector machines Other techniques: Boosting, Decision trees and forests, Deep neural networks

4. Image classification The big problems – Image classification eg this picture contains a parrot – Object detection eg in this box in the picture is a parrot Strategy – Generate features from image (there are many quite complex strategies) – Put in one or more classifiers 4

5. Image classification - scenes

6. Image classification - material

7. Classifiers Given a feature representation for images, how do we learn a model for distinguishing features from different classes? Zebra Non-zebra Decision boundary

8. Nearest Neighbor Classifier Assign label of nearest training data point to each test data point from Duda et al.

9. Classifiers Take a measurement x, predict a bit (yes/no; 1/-1; 1/0; etc) Strategies: – non-parametric nearest neighbor – probabilistic histogram logistic regression – decision boundary SVM

10. K-Nearest Neighbors For a new point, find the k closest points from training data Labels of the k points “vote” to classify k = 5

11. Histogram based classifiers Represent class-conditional densities with histogram Advantage: – estimates become quite good (with enough data!) Disadvantage: – Histogram becomes big with high dimension but maybe we can assume feature independence?

12. Example: Finding skin Skin has a very small range of (intensity independent) colours, and little texture – Compute an intensity-independent colour measure, check if colour is in this range, check if there is little texture (median filter) – See this as a classifier - we can set up the tests by hand, or learn them.

14. Histogram classifier for skin

15. Curse of dimension Can’t build a histogram based classifier for when feature space is high-dimensional – try R, G, B, and some texture features – Fails when there are too many histogram buckets

16. Distance functions for bags of features Euclidean distance: L1 distance: χ 2 distance: Histogram intersection (similarity): Hellinger kernel (similarity):

17. Linear classifiers Find linear function (hyperplane) to separate positive and negative examples Which hyperplane is best?

18. Support vector machines Find hyperplane that maximizes the margin between the positive and negative examples Margin Support vectors C. Burges, A Tutorial on Support Vector Machines for Pattern Recognition, Data Mining and Knowledge Discovery, 1998A Tutorial on Support Vector Machines for Pattern Recognition Distance between point and hyperplane: For support vectors, Therefore, the margin is 2 / ||w||

19. What if the data is not linearly separable? Demo: http://cs.stanford.edu/people/karpathy/svmjs/demohttp://cs.stanford.edu/people/karpathy/svmjs/demo Margin +1 0

20. Datasets that are linearly separable work out great: But what if the dataset is just too hard? We can map it to a higher-dimensional space: 0x 0 x 0 x x2x2 Nonlinear SVMs Slide credit: Andrew Moore

21. Φ: x → φ(x) Nonlinear SVMs General idea: the original input space can always be mapped to some higher- dimensional feature space where the training set is separable: Slide credit: Andrew Moore

22. Nonlinear SVMs The kernel trick: instead of explicitly computing the lifting transformation φ(x), define a kernel function K such that K(x, y) = φ(x) · φ(y) This gives a nonlinear decision boundary in the original feature space: C. Burges, A Tutorial on Support Vector Machines for Pattern Recognition, Data Mining and Knowledge Discovery, 1998A Tutorial on Support Vector Machines for Pattern Recognition

23. Nonlinear kernel: Example Consider the mapping x2x2

24. Polynomial kernel:

25. Gaussian kernel Also known as the radial basis function (RBF) kernel: The corresponding mapping φ(x) is infinite- dimensional!

26. Gaussian kernel SV’s

27. Kernels for bags of features Histogram intersection kernel: Hellinger kernel: Generalized Gaussian kernel: D can be L1, Euclidean, χ 2 distance, etc. J. Zhang, M. Marszalek, S. Lazebnik, and C. Schmid, Local Features and Kernels for Classifcation of Texture and Object Categories: A Comprehensive Study, IJCV 2007Local Features and Kernels for Classifcation of Texture and Object Categories: A Comprehensive Study

28. Summary: SVMs for image classification 1.Pick an image representation (in our case, bag of features) 2.Pick a kernel function for that representation 3.Compute the matrix of kernel values between every pair of training examples 4.Feed the kernel matrix into your favorite SVM solver to obtain support vectors and weights 5.At test time: compute kernel values for your test example and each support vector, and combine them with the learned weights to get the value of the decision function

29. What about multi-class SVMs? Unfortunately, there is no “definitive” multi-class SVM formulation In practice, we have to obtain a multi-class SVM by combining multiple two-class SVMs One vs. others – Traning: learn an SVM for each class vs. the others – Testing: apply each SVM to test example and assign to it the class of the SVM that returns the highest decision value One vs. one – Training: learn an SVM for each pair of classes – Testing: each learned SVM “votes” for a class to assign to the test example

30. SVMs: Pros and cons Pros – Many publicly available SVM packages: http://www.kernel-machines.org/software http://www.kernel-machines.org/software – Kernel-based framework is very powerful, flexible – SVMs work very well in practice, even with very small training sample sizes Cons – No “direct” multi-class SVM, must combine two-class SVMs – Computation, memory During training time, must compute matrix of kernel values for every pair of examples Learning can take a very long time for large-scale problems

31. SVMs for large-scale datasets Efficient linear solvers LIBLINEAR, PEGASOS LIBLINEARPEGASOS Explicit approximate embeddings: define an explicit mapping φ(x) such that φ(x) · φ(y) approximates K(x, y) and train a linear SVM on top of that embedding Random Fourier features for the Gaussian kernel (Rahimi and Recht, 2007) Embeddings for additive kernels, e.g., histogram intersection (Maji et al., 2013, Vedaldi and Zisserman, 2012)

32. Evaluating classifiers Always – train on training set, evaluate on test set test set performance might/should be worse than training set Options – Total error rate always less than 50% for two class – Receiver operating curve because we might use different thresholds – Class confusion matrix for multiclass

33. Receiver operating curve

35. Classifiers: Crucial Points Classifiers accept features, return decision – and are often trained using data Very mature training processes now – Always try standard methods first SVM kNN Logistic regression Evaluate with separate test data – look at total error rate – ROC – class confusion matrix – (occasionally) total risk typically when one has a loss model

36. Summary: Classifiers Nearest-neighbor and k-nearest-neighbor classifiers Support vector machines – Linear classifiers – Margin maximization – Non-separable case – The kernel trick – Multi-class SVMs – Large-scale SVMs There are so many other classifiers out there – Neural networks, boosting, decision trees/forests, …

37. Slide Credits Svetlana Lazebnik – UIUC David Forsyth - UIUC 37

38. Next class Object recognition Readings for next lecture: – Forsyth and Ponce chapter 17 Readings for today: – Forsyth and Ponce chapter 16 38

39. Questions 39