1. Computer Vision CS 543 / ECE 549 University of Illinois Derek Hoiem
03/11/10
Image Categorization
Computer Vision
CS 543 / ECE 549
University of Illinois
Derek Hoiem

2. Last classes
Object recognition: localizing an object instance in an image
Face recognition: matching one face image to another

3. Today’s class: categorization
Overview of image categorization
Representation
Image histograms
Classification
Important concepts in machine learning
What the classifiers are and when to use them

4. Image Categorization Training Training Labels Training Images
Image Features
Classifier Training
Trained Classifier

5. Image Categorization Training Testing Training Labels Training Images
Image Features
Classifier Training
Trained Classifier
Testing
Image Features
Trained Classifier
Prediction
Outdoor
Test Image

6. Part 1: Image features Training Training Labels Training Images
Classifier Training
Trained Classifier

7. General Principles of Representation
Coverage
Ensure that all relevant info is captured
Concision
Minimize number of features without sacrificing coverage
Directness
Ideal features are independently useful for prediction
Image Intensity

8. Right features depend on what you want to know
Shape: scene-scale, object-scale, detail-scale
2D form, shading, shadows, texture, linear perspective
Material properties: albedo, feel, hardness, …
Color, texture
Motion
Optical flow, tracked points
Distance
Stereo, position, occlusion, scene shape
If known object: size, other objects

9. Examples of cues
Color
Texture
Location
Perspective

10. Image representations
Image intensity
Histograms
Color, texture, SIFT keypoints, etc.

11. Image Representations: Histograms
Global histogram
Represent distribution of features
Color, texture, depth, …
Space Shuttle Cargo Bay
Images from Dave Kauchak

12. Image Representations: Histograms
Histogram: Probability or count of data in each bin
Joint histogram
Requires lots of data
Loss of resolution to avoid empty bins
Marginal histogram
Requires independent features
More data/bin than joint histogram
Images from Dave Kauchak

13. Image Representations: Histograms
Clustering
EASE Truss Assembly
Use the same cluster centers for all images
Space Shuttle Cargo Bay
Images from Dave Kauchak

14. Issue: How to Compare Histograms?
Bin-by-bin comparison
Sensitive to bin size.
Could use wider bins … … but at a loss of resolution
Cross-bin comparison
How much cross-bin influence is necessary/sufficient?

15. Computing histogram distance
Histogram intersection (assuming normalized histograms)
Chi-squared Histogram matching distance
Cars found by color histogram matching using chi-squared

16. Histograms: Implementation issues
Quantization
Grids: fast but only applicable with few dimensions
Clustering: slower but can quantize data in higher dimensions
Matching
Histogram intersection or Euclidean may be faster
Chi-squared often works better
Earth mover’s distance is good for when nearby bins represent similar values
Few Bins
Need less data
Coarser representation
Many Bins
Need more data
Finer representation

17. What kind of things do we compute histograms of?
Color
Texture (filter banks or HOG over regions)
L*a*b* color space
HSV color space

18. What kind of things do we compute histograms of?
Histograms of gradient
Visual words
SIFT – Lowe IJCV 2004

19. Bag of words model Extract features
E.g., compute SIFT keypoints for all training images

20. Bag of words model Extract features Learn “visual vocabulary”
E.g., cluster SIFT descriptors into 500 clusters using K-means

21. Bag of words model Extract features Learn “visual vocabulary”
Quantize features using visual vocabulary
Assign each descriptor to nearest cluster center

22. Bag of words model Extract features Learn “visual vocabulary”
Quantize features using visual vocabulary
Represent images by normalized counts of “visual words”
Compute histogram of word occurrences for each image

23. Image Categorization: Bag of Words
Training
Extract keypoints and descriptors for all training images
Cluster descriptors
Quantize descriptors using cluster centers to get “visual words”
Represent each image by normalized counts of “visual words”
Train classifier on labeled examples using histogram values as features
Testing
Extract keypoints/descriptors and quantize into visual words
Compute visual word histogram
Compute label or confidence using classifier

24. But what about layout?
All of these images have the same color histogram

25. Spatial pyramid
Compute histogram in each spatial bin

26. Image Categorization: BoW+SVM
Extract features
Learn “visual vocabulary”
Quantize features using visual vocabulary
Represent images by normalized counts of “visual words”
Train SVM on labeled examples using histogram values as features
For example, label images of indoor scenes as positive and images of outdoor scenes as negative. Then, train classifier based on features and labels.

27. Computing Features Compute Features over Image Quantize (Optional)
Choose Spatial Support
Compute Statistics of Features within Spatial Support
RGB Values
Quantized to 10 Levels
71%
29%
Histogram Bin Features

28. Things to remember about representation
Think about the right features for the problem
Coverage
Concision
Directness
Think about what features represent

29. Part 2: Classifiers Training Training Labels Training Images
Image Features
Classifier Training
Trained Classifier

30. Learning a classifier
Given some set features with corresponding labels, learn a function to predict the labels from the features x o x2 x1

31. Many classifiers to choose from
SVM
Neural networks
Naïve Bayes
Bayesian network
Logistic regression
Randomized Forests
Boosted Decision Trees
K-nearest neighbor
RBMs
Etc.
Which is the best one?

32. No Free Lunch Theorem
You can only get generalization through assumptions.

33. Bias-Variance Trade-off
MSE = bias2 + variance

34. Bias and Variance Error = bias2 + variance Few training examples
Complexity
Low Bias
High Variance
High Bias
Low Variance
Test Error
Few training examples
Many training examples

35. Choosing the trade-off
Need validation set
Validation set not same as test set
Complexity
Low Bias
High Variance
High Bias
Low Variance
Error
Test error
Training error

36. Effect of Training Size
Fixed classifier
Number of Training Examples
Error
Testing
Generalization Error
Training

37. How to measure complexity?
VC dimension
Upper bound on generalization error
Training error +
N: size of training set
h: VC dimension
: 1-probability

38. How to reduce variance? Choose a simpler classifier
Regularize the parameters
Get more training data

39. Risk Minimization
Margins x o x2 x1

40. The perfect classification algorithm
Objective function: solves what you want to solve
Parameterization: makes assumptions that fit the problem
Regularization: right level of regularization for amount of training data
Training algorithm: can find parameters that maximize objective on training set
Inference algorithm: can solve for objective function in evaluation

41. Generative vs. Discriminative Classifiers
Training
Maximize joint likelihood of data and labels
Assume (or learn) probability distribution and dependency structure
Can impose priors
Testing
P(y=1, x) / P(y=0, x) > t?
Examples
Foreground/background GMM
Naïve Bayes classifier
Bayesian network
Discriminative
Training
Learn to directly predict the labels from the data
Assume form of boundary
Margin maximization or parameter regularization
Testing
f(x) > t ; e.g., wTx > t
Examples
Logistic regression
SVM
Boosted decision trees

42. Classifiers Generative methods Discriminative methods Ensemble methods
Naïve Bayes
Bayesian Networks
Gaussian mixture for both classes
Discriminative methods
Logistic Regression
Linear SVM
Kernelized SVM
Ensemble methods
Randomized Forests
Boosted Decision Trees
Instance based
K-nearest neighbor
Unsupervised
Kmeans

43. Generative Classifier: Naïve Bayes
Objective
Parameterization
Regularization
Training
Inference y x1 x2 x3

44. Using Naïve Bayes Simple thing to try for categorical data
Very fast to train/test

45. Classifiers: Logistic Regression
Objective
Parameterization
Regularization
Training
Inference x o x2 x1

46. Using Logistic Regression
Quick, simple classifier (try it first)
Use L2 or L1 regularization
L1 does feature selection and is robust to irrelevant features

47. Classifiers: Linear SVM
Objective
Parameterization
Regularization
Training
Inference x o x2 x1

48. Classifiers: Linear SVM
Objective
Parameterization
Regularization
Training
Inference x o x2 x1

49. Classifiers: Linear SVM
Objective
Parameterization
Regularization
Training
Inference x o x2 x1

50. Classifiers: Kernelized SVM
Objective
Parameterization
Regularization
Training
Inference x o x o x2

51. Using SVMs Good general purpose classifier Choosing kernel
Generalization depends on margin, so works well with many weak features
No feature selection
Usually requires some parameter tuning
Choosing kernel
Linear: fast training/testing – start here
RBF: related to neural networks, nearest neighbor
Chi-squared, histogram intersection: good for histograms (but slower, esp. chi-squared)
Can learn a kernel function

52. Classifiers: Decision Trees
Objective
Parameterization
Regularization
Training
Inference x o x2 x1

53. Ensemble Methods: Boosting
figure from Friedman et al. 2000

54. Boosted Decision Trees
High in
Image?
Gray?
Yes No
Yes No
Smooth?
Green?
High in
Image?
Many Long
Lines? …
Yes
Yes No
Yes No
Yes No No
Blue?
Very High
Vanishing
Point?
Yes No
Yes No
P(label | good segment, data)
Ground Vertical Sky
[Collins et al. 2002]

55. Using Boosted Decision Trees
Flexible: can deal with both continuous and categorical variables
How to control bias/variance trade-off
Size of trees
Number of trees
Boosting trees often works best with a small number of well-designed features
Boosting “stubs” can give a fast classifier

56. K-nearest neighbor Objective Parameterization Regularization Training
Inference x o x2 x1 + +

57. 1-nearest neighbor x o x2 x1 + +

58. 3-nearest neighbor x o x2 x1 + +

59. 5-nearest neighbor x o x2 x1 + +

60. Using K-NN Simple, so another good one to try first
With infinite examples, 1-NN provably has error that is at most twice Bayes optimal error

61. Clustering (unsupervised)x2 + x1 x o x1

62. What to remember about classifiers
No free lunch: machine learning algorithms are tools, not dogmas
Try simple classifiers first
Better to have smart features and simple classifiers than simple features and smart classifiers
Use increasingly powerful classifiers with more training data (bias-variance tradeoff)

63. Next class
Object category detection overview

64. Some Machine Learning References
General
Tom Mitchell, Machine Learning, McGraw Hill, 1997
Christopher Bishop, Neural Networks for Pattern Recognition, Oxford University Press, 1995
Adaboost
Friedman, Hastie, and Tibshirani, “Additive logistic regression: a statistical view of boosting”, Annals of Statistics, 2000
SVMs