Object Recognition Szeliski Chapter 14

Recognition

Using context (Russell, Torralba, Liu et al. 2007). Simultaneous recognition and segmentation (Shotton, Winn, Rother et al. 2009) Location recognition (Philbin, Chum, Isard et al. 2007) Recognition

Slides from Rick Szeliski Recognition problems What is it? Object and scene recognition Who is it? Identity recognition Where is it? Object detection What are they doing? Activities All of these are classification problems Choose one class from a list of possible candidates Recognition Slides from Rick Szeliski

Slides from Rick Szeliski What is recognition? A different taxonomy from [Csurka et al. 2006]: Recognition Where is this particular object? Categorization What kind of object(s) is(are) present? Content-based image retrieval Find me something that looks similar Detection Locate all instances of a given class Recognition Slides from Rick Szeliski

Slides from Rick Szeliski Readings Weakly Supervised Scale-Invariant Learning of Models for Visual Recognition Fergus, R. , Perona, P. and Zisserman, A. International Journal of Computer Vision, Vol. 71(3), 273-303, March 2007 MIT course http://people.csail.mit.edu/torralba/courses/6.870/6.870.recognition.htm Recognition Slides from Rick Szeliski

Slides from Rick Szeliski Sources Steve Seitz, CSE 455/576, previous quarters Fei-Fei, Fergus, Torralba, CVPR’2007 course Efros, CMU 16-721 Learning in Vision Freeman, MIT 6.869 Computer Vision: Learning Linda Shapiro, CSE 576, Spring 2007 Recognition Slides from Rick Szeliski

Today’s lecture Object Detection Known object recognition [Lowe] Bag of keypoints [Csurka etc.] Location recognition [Schindler et al.] Deformable object/category recognition [Fergus et al.] Recognition by segmentation Recognition

Object Detection How to recognize each person in this image? Every possible sub-window? Effective special-purpose detectors: rapidly find likely regions where particular objects might occur. How to recognize each person in this image? Recognition

Object Detection Face Detection General object detection More successful Built in digital cameras to enhance auto-focus Video conference to control pan-tilt heads … General object detection Pedestrian Cars. Recognition

Face Detection Every pixel? Scale? Tutorials Too slow in practice http://people.csail.mit.edu/torralba/shortCourseRLOC/ General detection and recognition http://vision.ai.uiuc.edu/mhyang/face-detection-survey.html Face recognition Recognition

Face Recognition and Detection Face detection You can ask people to see what they come up with How to tell if a face is present? CSE 576, Spring 2008 Face Recognition and Detection 12

Face Recognition and Detection Skin detection skin Skin pixels have a distinctive range of colors Corresponds to region(s) in RGB color space Skin classifier A pixel X = (R,G,B) is skin if it is in the skin (color) region How to find this region? CSE 576, Spring 2008 Face Recognition and Detection

Face Recognition and Detection Skin detection Learn the skin region from examples Manually label skin/non pixels in one or more “training images” Plot the training data in RGB space skin pixels shown in orange, non-skin pixels shown in gray some skin pixels may be outside the region, non-skin pixels inside. CSE 576, Spring 2008 Face Recognition and Detection

Face Recognition and Detection Skin classifier Given X = (R,G,B): how to determine if it is skin or not? Nearest neighbor find labeled pixel closest to X Find plane/curve that separates the two classes popular approach: Support Vector Machines (SVM) Data modeling fit a probability density/distribution model to each class CSE 576, Spring 2008 Face Recognition and Detection

Face Recognition and Detection Probability X is a random variable P(X) is the probability that X achieves a certain value called a PDF probability distribution/density function a 2D PDF is a surface 3D PDF is a volume continuous X discrete X CSE 576, Spring 2008 Face Recognition and Detection

Probabilistic skin classification Model PDF / uncertainty Each pixel has a probability of being skin or not skin Skin classifier Given X = (R,G,B): how to determine if it is skin or not? Choose interpretation of highest probability Where do we get and ? CSE 576, Spring 2008 Face Recognition and Detection

Learning conditional PDF’s We can calculate P(R | skin) from a set of training images It is simply a histogram over the pixels in the training images each bin Ri contains the proportion of skin pixels with color Ri This doesn’t work as well in higher-dimensional spaces. Why not? Approach: fit parametric PDF functions common choice is rotated Gaussian center covariance CSE 576, Spring 2008 Face Recognition and Detection

Learning conditional PDF’s We can calculate P(R | skin) from a set of training images But this isn’t quite what we want Why not? How to determine if a pixel is skin? We want P(skin | R) not P(R | skin) How can we get it? CSE 576, Spring 2008 Face Recognition and Detection

Face Recognition and Detection Bayes rule what we measure (likelihood) domain knowledge (prior) In terms of our problem: what we want (posterior) normalization term What can we use for the prior P(skin)? Domain knowledge: P(skin) may be larger if we know the image contains a person For a portrait, P(skin) may be higher for pixels in the center Learn the prior from the training set. How? CSE 576, Spring 2008 Face Recognition and Detection P(skin) is proportion of skin pixels in training set

Face Recognition and Detection Bayesian estimation likelihood posterior (unnormalized) Bayesian estimation Goal is to choose the label (skin or ~skin) that maximizes the posterior ↔ minimizes probability of misclassification this is called Maximum A Posteriori (MAP) estimation CSE 576, Spring 2008 Face Recognition and Detection

Skin detection results CSE 576, Spring 2008 Face Recognition and Detection

General classification This same procedure applies in more general circumstances More than two classes More than one dimension Example: face detection Here, X is an image region dimension = # pixels each face can be thought of as a point in a high dimensional space H. Schneiderman, T. Kanade. "A Statistical Method for 3D Object Detection Applied to Faces and Cars". CVPR 2000 CSE 576, Spring 2008 Face Recognition and Detection

Face Detection Feature-based Template-based Appearance-based Distinctive image features: eyes, nose, mouth Geometrical arrangement Template-based Active shape model (ASM), active appearance model (AAM) Requires good initialization near a real face Appearance-based Search for likely candidates Refine using cascade of more expensive but selective detection algorithm Rely heavily on training classifiers using labeled faces Recognition

Slides from Rick Szeliski Recognition Slides from Rick Szeliski

CVPR 2007 Minneapolis, Short Course, June 17 Recognizing and Learning Object Categories: Year 2007 Li Fei-Fei, Princeton Rob Fergus, MIT Antonio Torralba, MIT (see other slide deck)

Slides from Rick Szeliski Today’s lecture Known object recognition [Lowe] Bag of keypoints [Csurka etc.] Location recognition [Schindler et al.] Deformable object/category recognition [Fergus et al.] Recognition by segmentation Recognition Slides from Rick Szeliski

Single object recognition Slides from Rick Szeliski

Single object recognition Lowe, et al. 1999, 2003 Mahamud and Herbert, 2000 Ferrari, Tuytelaars, and Van Gool, 2004 Rothganger, Lazebnik, and Ponce, 2004 Moreels and Perona, 2005 … Recognition Slides from Rick Szeliski

Planar object recognition [Lowe] Use SIFT features Verify affine (or homography) geometric alignment Recognition Slides from Rick Szeliski

Planar object recognition [Lowe] Use SIFT features Verify affine (or homography) geometric alignment Recognition Slides from Rick Szeliski

3D object recognition [Lowe] Extract object outlines with background subtraction Recognition Slides from Rick Szeliski

3D object recognition [Lowe] Use 3 matches to recognize Use additional matches for verification Tolerant to occlusions Recognition Slides from Rick Szeliski

Feature-based recognition How can we scale to millions of objects? Comparison to all stored objects/features is infeasible. Answer: quantize features into words [Csurka et al. 04] use information retrieval (inverted index) use metric tree for faster quantization (NN) [Nister & Stewenius 05] Recognition Slides from Rick Szeliski

Slides from Rick Szeliski Today’s lecture Known object recognition [Lowe] Bag of keypoints [Csurka etc.] Location recognition [Schindler et al.] Deformable object/category recognition [Fergus et al.] Recognition by segmentation Recognition Slides from Rick Szeliski

Part 1: Bag-of-words models CVPR 2007 Minneapolis, Short Course, June 17 (see other slide deck) Part 1: Bag-of-words models by Li Fei-Fei (Princeton)

Slides from Rick Szeliski Today’s lecture Known object recognition [Lowe] Bag of keypoints [Csurka etc.] Location recognition [Schindler et al.] Deformable object/category recognition [Fergus et al.] Recognition by segmentation Recognition Slides from Rick Szeliski

How to scale to 106s of images? Make “word” generation even more efficient: “Vocabulary tree” Recognition Slides from Rick Szeliski

Scalable Recognition with a Vocabulary Tree David Nistér, Henrik Stewénius Recognition Slides from Rick Szeliski

Slides from Rick Szeliski Vocabulary Tree Recognition Slides from Rick Szeliski

Slides from Rick Szeliski We then run k-means on the descriptor space. In this setting, k defines what we call the branch-factor of the tree, which indicates how fast the tree branches. In this illustration, k is three. We then run k-means again, recursively on each of the resulting quantization cells. This defines the vocabulary tree, which is essentially a hierarchical set of cluster centers and their corresponding Voronoi regions. We typically use a branch-factor of 10 and six levels, resulting in a million leaf nodes. We lovingly call this the Mega-Voc. Recognition Slides from Rick Szeliski

Slides from Rick Szeliski Performance One of the reasons we managed to take the system this far is that we have worked with rigorous performance testing against a database with ground truth. We now have a benchmark database with over 10000 images grouped in sets of four images of the same object. We query on one of the images in the group and see how many of the other images in the group make it to the top of the query. In the paper, we have tested this up to a million images, by embedding the ground truth database into a database encompassing all the frames of 10 feature length movies. Recognition Slides from Rick Szeliski

Slides from Rick Szeliski http://vis.uky.edu/~stewe/ukbench/ We have just posted the whole 10000 image test set on the web for everyone to use. Just Google my webpage and you will find it under the title Recognition Benchmark. Recognition Slides from Rick Szeliski

Slides from Rick Szeliski Location Recognition Can we apply this to recognizing your location from a cell-phone photo? Recognition Slides from Rick Szeliski

City-Scale Location Recognition Grant Schindler, Matthew Brown, and Richard Szeliski CVPR’2007

Slides from Rick Szeliski The Problem Recognition Slides from Rick Szeliski

Slides from Rick Szeliski Main idea Find N-best matches in vocabulary tree Recognition Slides from Rick Szeliski

Slides from Rick Szeliski Other ideas Use only informative features (ignore trees…) Integrate matches with adjacent (streetside) neighbors Recognition Slides from Rick Szeliski

Slides from Rick Szeliski Today’s lecture Known object recognition [Lowe] Bag of keypoints [Csurka etc.] Location recognition [Schindler et al.] Deformable object/category recognition [Fergus et al.] Recognition by segmentation Recognition Slides from Rick Szeliski

Part 2: part-based models CVPR 2007 Minneapolis, Short Course, June 17 (see other slide deck) Part 2: part-based models by Rob Fergus (MIT)

Slides from Rick Szeliski Today’s lecture Known object recognition [Lowe] Bag of keypoints [Csurka etc.] Location recognition [Schindler et al.] Deformable object/category recognition [Fergus et al.] Recognition by segmentation Recognition Slides from Rick Szeliski

Part 4: Combined segmentation and recognition CVPR 2007 Minneapolis, Short Course, June 17 Part 4: Combined segmentation and recognition by Rob Fergus (MIT)

Slides from Rick Szeliski Aim Given an image and object category, segment the object Object Category Model Segmentation Cow Image Segmented Cow Segmentation should (ideally) be shaped like the object e.g. cow-like obtained efficiently in an unsupervised manner able to handle self-occlusion Recognition Slides from Rick Szeliski Slide from Kumar ‘05

Implicit Shape Model - Liebe and Schiele, 2003 Matched Codebook Entries Probabilistic Voting Interest Points Voting Space (continuous) Segmentation Refined Hypotheses (uniform sampling) Backprojected Hypotheses Backprojection of Maxima Recognition Slides from Rick Szeliski

Other topics: context (scenes) Antonio Torralba, Contextual Priming for Object Detection, IJCV(53), No. 2, July 2003, pp. 169-191 Recognition Slides from Rick Szeliski

Slides from Rick Szeliski New work: tiny images Recognition Slides from Rick Szeliski

Datasets and object collections CVPR 2007 Minneapolis, Short Course, June 17 (see other slide deck) Datasets and object collections

Summary of object recognition Known object recognition [Lowe] Bag of keypoints [Csurka etc.] Location recognition [Schindler et al.] Deformable object/category recognition [Fergus et al.] Recognition by segmentation Context and scenes Recognition Slides from Rick Szeliski