Window-based models for generic object detection Mei-Chen Yeh 04/24/2012

Object Detection Find the location of an object if it appear in an image –Does the object appear? –Where is it?

Viola-Jones face detector P. Viola and M. J. Jones. Robust Real-Time Face Detection. IJCV 2004.

Viola-Jones Face Detector: Results

Paul Viola, ICCV tutorial Viola-Jones Face Detector: Results

A successful application

Consumer application: iPhoto Slide credit: Lana Lazebnik

Consumer application: iPhoto 2009 Things iPhoto thinks are faces Slide credit: Lana Lazebnik

Consumer application: iPhoto 2009 Can be trained to recognize pets! Slide credit: Lana Lazebnik

Challenges Slide credit: Fei-Fei Li Michelangelo view point variation illumination occlusion Magritte, 1957 scale

Challenges Xu, Beihong 1943 deformation Slide credit: Fei-Fei Li Klimt, 1913 background clutter

Basic framework Build/train object model –Choose a representation –Learn or fit parameters of model / classifier Generate candidates in new image Score the candidates

Basic framework Build/train object model –Choose a representation –Learn or fit parameters of model / classifier Generate candidates in new image Score the candidates

Window-based models Building an object model Face/non-face Classifier Yes, face. No, not a face. Given the representation, train a binary classifier

Basic framework Build/train object model –Choose a representation –Learn or fit parameters of model / classifier Generate candidates in new image Score the candidates

face/non-face Classifier Scans the detector at multiple locations and scales Window-based models Generate and score candidates

Window-based object detection Car/non-car Classifier Feature extraction Training examples Training: 1. Obtain training data 2. Define features 3. Define classifier Given new image: 1. Slide window 2. Score by classifier

Viola-Jones detection approach Viola and Jones’ face detection algorithm –The first object detection framework to provide competitive object detection rates in real-time –Implemented in OpenCV Components –Features Haar-features Integral image –Learning Boosting algorithm –Cascade method

Haar-features (1) The difference between pixels’ sum of the white and black areas

Haar-features (2) Capture the face symmetry

A 24x24 detection window Four types of haar features Type A Haar-features (3) Can be extracted at any location with any scale!

Haar-features (4) Too many features! –location, scale, type –180,000+ possible features associated with each 24 x 24 window Not all of them are useful! Speed-up strategy –Fast calculation of haar-features –Selection of good features 24 AdaBoost

Integral image (1) Sum of pixel values in the blue area Example: Image Integral image Time complexity?

Integral image (2) ab c d a = sum(1) b = sum(1+2) c = sum(1+3) d = sum( ) Sum(4) = ? 4 d + a – b – c Four-point calculation! A, B: 2 rectangles => C: 3 rectangles => D: 4 rectangles => 6-point 8-point 9-point

Feature selection A very small number of features can be combined to from an effective classifier! Example: The 1 st and 2 nd features selected by AdaBoost

Feature selection A weak classifier h f1f1 f2f2 f 1 > θ (a threshold) => Face! f 2 ≤ θ (a threshold) => Not a Face! h = 1 if f i > θ 0 otherwise

Feature selection Idea: Combining several weak classifiers to generate a strong classifier α1α1 α2α2 α3α3 αTαT …… α 1 h 1 + α 2 h 2 + α 3 h 3 + … + α T h T ><>< T thresold weak classifier (feature, threshold) h 1 = 1 or 0 ~ performance of the weak classifier on the training set

Feature selection Training Dataset –4916 face images –non-face images cropped from 9500 images non-face images positive samplesnegative samples

AdaBoost Each training sample may have different importance! Focuses more on previously misclassified samples –Initially, all samples are assigned equal weights –Weights may change at each boosting round misclassified samples => increase their weights correctly classified samples => decrease their weights

Boosting illustration: 2D case Weak Classifier 1 Slide credit: Paul Viola

Boosting illustration Weights Increased

Boosting illustration Weak Classifier 2

Boosting illustration Weights Increased

Boosting illustration Weak Classifier 3

Boosting illustration Final classifier is a combination of weak classifiers

Viola-Jones detector: AdaBoost Want to select the single rectangle feature and threshold that best separates positive (faces) and negative (non- faces) training examples, in terms of weighted error. Outputs of a possible rectangle feature on faces and non-faces. … Resulting weak classifier: For next round, reweight the examples according to errors, choose another filter/threshold combo. Kristen Grauman

AdaBoost decreased increased fifi Initial weights for each data point -∞∞ misclassified ~ error rate error↘ α↗

AdaBoost ……

Learning the classifier Initialize equal weights to training samples For T rounds –normalize the weights –select the best weak classifier in terms of the weighted error –update the weights (raise weights to misclassified samples) Linearly combine these T weak classifiers to form a strong classifier

AdaBoost Algorithm Start with uniform weights on training examples Evaluate weighted error for each feature, pick best. Re-weight the examples: Incorrectly classified -> more weight Correctly classified -> less weight Final classifier is combination of the weak ones, weighted according to error they had. Freund & Schapire 1995 {x 1,…x n } For T rounds

First two features selected Feature Selection: Results

Boosting: pros and cons Advantages of boosting –Integrates classification with feature selection –Complexity of training is linear in the number of training examples –Flexibility in the choice of weak learners, boosting scheme –Testing is fast –Easy to implement Disadvantages –Needs many training examples –Often found not to work as well as an alternative discriminative classifier, support vector machine (SVM) especially for many-class problems Slide credit: Lana Lazebnik

Viola-Jones detection approach Viola and Jones’ face detection algorithm –The first object detection framework to provide competitive object detection rates in real-time –Implemented in OpenCV Components –Features Haar-features Integral image –Learning Boosting algorithm –Cascade method

Even if the filters are fast to compute, each new image has a lot of possible windows to search. How to make the detection more efficient?

Cascade method Strong Classifier = (α 1 h 1 + α 2 h 2 ) + (…)+ (…+ α T h T ) 123 ><>< T thresold Most windows contain no face! Rejects negative windows in an early stage!

Viola-Jones detector: summary Train with 5K positives, 350M negatives Real-time detector using 38 layer cascade 6061 features in all layers Implementation available in OpenCV Faces Non-faces Train cascade of classifiers with AdaBoost Selected features, thresholds, and weights New image Apply to each subwindow Kristen Grauman

Questions What other categories are amenable to window-based representation? Can the integral image technique be applied to compute histograms? Alternatives to sliding-window-based approaches?