Part 3: discriminative methods Antonio Torralba

Overview of section Object detection with classifiers Boosting –Gentle boosting –Weak detectors –Object model –Object detection Multiclass object detection

Discriminative methods Object detection and recognition is formulated as a classification problem. Bag of image patches Decision boundary … and a decision is taken at each window about if it contains a target object or not. Computer screen Background In some feature space Where are the screens? The image is partitioned into a set of overlapping windows

(The lousy painter) Discriminative vs. generative x = data Generative model x = data Discriminative model x = data Classification function (The artist)

Discriminative methods 10 6 examples Nearest neighbor Shakhnarovich, Viola, Darrell 2003 Berg, Berg, Malik 2005 … Neural networks LeCun, Bottou, Bengio, Haffner 1998 Rowley, Baluja, Kanade 1998 … Support Vector Machines and Kernels Conditional Random Fields McCallum, Freitag, Pereira 2000 Kumar, Hebert 2003 … Guyon, Vapnik Heisele, Serre, Poggio, 2001 …

Formulation: binary classification Formulation +1 x1x1 x2x2 x3x3 xNxN … … x N+1 x N+2 x N+M ??? … Training data: each image patch is labeled as containing the object or background Test data Features x = Labels y = Where belongs to some family of functions Classification function Minimize misclassification error (Not that simple: we need some guarantees that there will be generalization)

Overview of section Object detection with classifiers Boosting –Gentle boosting –Weak detectors –Object model –Object detection Multiclass object detection

A simple object detector with Boosting Download Toolbox for manipulating dataset Code and dataset Matlab code Gentle boosting Object detector using a part based model Dataset with cars and computer monitors

A simple algorithm for learning robust classifiers –Freund & Shapire, 1995 –Friedman, Hastie, Tibshhirani, 1998 Provides efficient algorithm for sparse visual feature selection –Tieu & Viola, 2000 –Viola & Jones, 2003 Easy to implement, not requires external optimization tools. Why boosting?

Defines a classifier using an additive model: Boosting Strong classifier Weak classifier Weight Features vector

Defines a classifier using an additive model: We need to define a family of weak classifiers Boosting Strong classifier Weak classifier Weight Features vector from a family of weak classifiers

Each data point has a class label: w t =1 and a weight: +1 ( ) -1 ( ) y t = Boosting It is a sequential procedure: x t=1 x t=2 xtxt

Toy example Weak learners from the family of lines h => p(error) = 0.5 it is at chance Each data point has a class label: w t =1 and a weight: +1 ( ) -1 ( ) y t =

Toy example This one seems to be the best Each data point has a class label: w t =1 and a weight: +1 ( ) -1 ( ) y t = This is a ‘weak classifier’: It performs slightly better than chance.

Toy example We set a new problem for which the previous weak classifier performs at chance again Each data point has a class label: w t w t exp{-y t H t } We update the weights: +1 ( ) -1 ( ) y t =

Toy example We set a new problem for which the previous weak classifier performs at chance again Each data point has a class label: w t w t exp{-y t H t } We update the weights: +1 ( ) -1 ( ) y t =

Toy example We set a new problem for which the previous weak classifier performs at chance again Each data point has a class label: w t w t exp{-y t H t } We update the weights: +1 ( ) -1 ( ) y t =

Toy example We set a new problem for which the previous weak classifier performs at chance again Each data point has a class label: w t w t exp{-y t H t } We update the weights: +1 ( ) -1 ( ) y t =

Toy example The strong (non- linear) classifier is built as the combination of all the weak (linear) classifiers. f1f1 f2f2 f3f3 f4f4

Boosting Different cost functions and minimization algorithms result is various flavors of Boosting In this demo, I will use gentleBoosting: it is simple to implement and numerically stable.

Overview of section Object detection with classifiers Boosting –Gentle boosting –Weak detectors –Object model –Object detection Multiclass object detection

Boosting Boosting fits the additive model by minimizing the exponential loss Training samples The exponential loss is a differentiable upper bound to the misclassification error.

Exponential loss Squared error Exponential loss yF(x) = margin Misclassification error Loss Squared error Exponential loss

Boosting Sequential procedure. At each step we add For more details: Friedman, Hastie, Tibshirani. “Additive Logistic Regression: a Statistical View of Boosting” (1998) to minimize the residual loss input Desired output Parameters weak classifier

gentleBoosting For more details: Friedman, Hastie, Tibshirani. “Additive Logistic Regression: a Statistical View of Boosting” (1998) We chose that minimizes the cost: At each iterations we just need to solve a weighted least squares problem Weights at this iteration At each iteration: Instead of doing exact optimization, gentle Boosting minimizes a Taylor approximation of the error:

Weak classifiers The input is a set of weighted training samples (x,y,w) Regression stumps: simple but commonly used in object detection. Four parameters: b=E w (y [x>  ]) a=E w (y [x<  ]) x f m (x)  fitRegressionStump.m

gentleBoosting.m function classifier = gentleBoost(x, y, Nrounds) … for m = 1:Nrounds fm = selectBestWeakClassifier(x, y, w); w = w.* exp(- y.* fm); % store parameters of fm in classifier … end Solve weighted least-squares Re-weight training samples Initialize weights w = 1

Demo gentleBoosting > demoGentleBoost.m Demo using Gentle boost and stumps with hand selected 2D data:

Flavors of boosting AdaBoost (Freund and Shapire, 1995) Real AdaBoost (Friedman et al, 1998) LogitBoost (Friedman et al, 1998) Gentle AdaBoost (Friedman et al, 1998) BrownBoosting (Freund, 2000) FloatBoost (Li et al, 2002) …

Overview of section Object detection with classifiers Boosting –Gentle boosting –Weak detectors –Object model –Object detection Multiclass object detection

From images to features: Weak detectors We will now define a family of visual features that can be used as weak classifiers (“weak detectors”) Takes image as input and the output is binary response. The output is a weak detector.

Weak detectors Textures of textures Tieu and Viola, CVPR 2000 Every combination of three filters generates a different feature This gives thousands of features. Boosting selects a sparse subset, so computations on test time are very efficient. Boosting also avoids overfitting to some extend.

Weak detectors Haar filters and integral image Viola and Jones, ICCV 2001 The average intensity in the block is computed with four sums independently of the block size.

Weak detectors Other weak detectors: Carmichael, Hebert 2004 Yuille, Snow, Nitzbert, 1998 Amit, Geman 1998 Papageorgiou, Poggio, 2000 Heisele, Serre, Poggio, 2001 Agarwal, Awan, Roth, 2004 Schneiderman, Kanade 2004 …

Weak detectors Part based: similar to part-based generative models. We create weak detectors by using parts and voting for the object center location Car model Screen model These features are used for the detector on the course web site.

Weak detectors First we collect a set of part templates from a set of training objects. Vidal-Naquet, Ullman (2003) …

Weak detectors We now define a family of “weak detectors” as: == Better than chance *

Weak detectors We can do a better job using filtered images Still a weak detector but better than before * * = = =

Training First we evaluate all the N features on all the training images. Then, we sample the feature outputs on the object center and at random locations in the background:

Representation and object model … 410 Selected features for the screen detector 123 … 100 Lousy painter

Representation and object model Selected features for the car detector … …

Overview of section Object detection with classifiers Boosting –Gentle boosting –Weak detectors –Object model –Object detection Multiclass object detection

Example: screen detection Feature output

Example: screen detection Feature output Thresholded output Weak ‘detector’ Produces many false alarms.

Example: screen detection Feature output Thresholded output Strong classifier at iteration 1

Example: screen detection Feature output Thresholded output Strong classifier Second weak ‘detector’ Produces a different set of false alarms.

Example: screen detection + Feature output Thresholded output Strong classifier Strong classifier at iteration 2

Example: screen detection + … Feature output Thresholded output Strong classifier Strong classifier at iteration 10

Example: screen detection + … Feature output Thresholded output Strong classifier Adding features Final classification Strong classifier at iteration 200

Demo > runDetector.m Demo of screen and car detectors using parts, Gentle boost, and stumps:

Probabilistic interpretation Generative model Discriminative (Boosting) model. Boosting is fitting an additive logistic regression model: It can be a set of arbitrary functions of the image This provides a great flexibility, difficult to beat by current generative models. But also there is the danger of not understanding what are they really doing.

Weak detectors Generative model f i, P i gigi Image Feature Part template Relative position wrt object center Discriminative (Boosting) model. Boosting is fitting an additive logistic regression model:

Object models Invariance: search strategy Part based f i, P i gigi Here, invariance in translation and scale is achieved by the search strategy: the classifier is evaluated at all locations (by translating the image) and at all scales (by scaling the image in small steps). The search cost can be reduced using a cascade.

We want the complexity of the 3 features classifier with the performance of the 100 features classifier: Cascade of classifiers Fleuret and Geman 2001, Viola and Jones 2001 Recall Precision 0% 100% 3 features 30 features 100 features Select a threshold with high recall for each stage. We increase precision using the cascade

Single category object detection and the “Head in the coffee beans problem”

“Head in the coffee beans problem” Can you find the head in this image?

Overview of section Object detection with classifiers Boosting –Gentle boosting –Weak detectors –Object model –Object detection Multiclass object detection

Multiclass object detection benefits from: Contextual relationships between objects Transfer between classes by sharing features Studying the multiclass problem, we can build detectors that are: more efficient, that generalize better, and more robust

Context 2 What do you think are the hidden objects? 1

Context What do you think are the hidden objects? Even without local object models, we can make reasonable detections!

Context: relationships between objects Detect first simple objects (reliable detectors) that provide strong contextual constraints to the target (screen -> keyboard -> mouse)

Context Murphy, Torralba & Freeman (NIPS 03) Use global context to predict presence and location of objects O p1,c1 v p1,c1 O pN,c1 v pN,c1... O p1,c2 v p1,c2 O pN,c2 v pN,c2... Class 1 Class 2 E1 E2 S c2 max V c1 max V X1 X2 vgvg Keyboards

Context Fink & Perona (NIPS 03) Use output of boosting from other objects at previous iterations as input into boosting for this iteration

Context Hoiem, Efros, Hebert (ICCV 05) Boosting used for combining local and contextual features:

Features Parts Objects Scene Context (generative model) Sudderth, Torralba, Freeman, Willsky (ICCV 2005). Context Sharing

Some references on context Strat & Fischler (PAMI 91) Torralba & Sinha (ICCV 01), Torralba (IJCV 03) Fink & Perona (NIPS 03) Murphy, Torralba & Freeman (NIPS 03) Kumar and M. Hebert (NIPS 04) Carbonetto, Freitas & Barnard (ECCV 04) He, Zemel & Carreira-Perpinan (CVPR 04) Sudderth, Torralba, Freeman, Wilsky (ICCV 05) Hoiem, Efros, Hebert (ICCV 05) … With a mixture of generative and discriminative approaches

Sharing features Shared features achieve the same performances with five times less training data. We train 12 classifies with shared features Torralba, Murphy & Freeman (CVPR 04)

Sharing features shared weak detector Torralba, Murphy & Freeman (CVPR 04) Strength of feature response Part template

Sharing representations Bart and Ullman, 2004 For a new class, use only features similar to features that where good for other classes: Proposed Dog features

Some references on multiclass Caruana 1997 Schapire, Singer, 2000 Thrun, Pratt 1997 Krempp, Geman, Amit, 2002 E.L.Miller, Matsakis, Viola, 2000 Mahamud, Hebert, Lafferty, 2001 Fink 2004 LeCun, Huang, Bottou, 2004 Holub, Welling, Perona, 2005 …

Summary Many techniques are used for training discriminative models that I have not mention here: Conditional random fields Kernels for object recognition Learning object similarities …