11/26/2015 Copyright G.D. Hager Class 2 - Schedule 1.Optical Illusions 2.Lecture on Object Recognition 3.Group Work 4.Sports Videos 5.Short Lecture on Rigid Transformations 6.Lab time

11/26/2015 Copyright G.D. Hager Object Recognition Techniques

11/26/2015 Copyright G.D. Hager Li Fei-Fei, UIUC Rob Fergus, MIT Antonio Torralba, MIT Recognizing and Learning Object Categories ICCV 2005 Beijing, Short Course, Oct 15

11/26/2015 Copyright G.D. Hager

11/26/2015 Copyright G.D. Hager perceptible vision materialthing

11/26/2015 Copyright G.D. Hager

11/26/2015 Copyright G.D. Hager

11/26/2015 Copyright G.D. Hager Plato said… Ordinary objects are classified together if they `participate' in the same abstract Form, such as the Form of a Human or the Form of Quartz. Forms are proper subjects of philosophical investigation, for they have the highest degree of reality. Ordinary objects, such as humans, trees, and stones, have a lower degree of reality than the Forms. Fictions, shadows, and the like have a still lower degree of reality than ordinary objects and so are not proper subjects of philosophical enquiry.

11/26/2015 Copyright G.D. Hager Bruegel, 1564

11/26/2015 Copyright G.D. Hager How many object categories are there? Biederman 1987

11/26/2015 Copyright G.D. Hager Problems of Computer Vision: Recognition Given a database of objects and an image determine what, if any of the objects are present in the image.

11/26/2015 Copyright G.D. Hager Problems of Computer Vision: Recognition Given a database of objects and an image determine what, if any of the objects are present in the image.

11/26/2015 Copyright G.D. Hager Problems of Computer Vision: Recognition Given a database of objects and an image determine what, if any of the objects are present in the image.

11/26/2015 Copyright G.D. Hager Object Recognition: The Problem Given: A database D of “known” objects and an image I: 1. Determine which (if any) objects in D appear in I 2. Determine the pose (rotation and translation) of the object Segmentation (where is it 2D) Recognition (what is it) The object recognition conundrum Pose Est. (where is it 3D)

11/26/2015 Copyright G.D. Hager Object Recognition Issues How general is the problem? –2D vs. 3D –range of viewing conditions –available context –segmentation cues What sort of data is best suited to the problem? –local 2D features –3D surfaces –images How many objects are involved? –small: brute force search –large: ??

11/26/2015 Copyright G.D. Hager Object Recognition Approaches Geometry-based –Interpretation trees: use features compute “local constraints” valid under Euclidean or similarity group –Invariants: use features compute “global indices” that do not change over viewing conditions (i.e. invariant in the projective group) Image-based: –store information about views and match to views intensities histograms Semi-local: –use features detected using a stable (but not invariant) interest operator –use stable (but not invariant) measures on groups of features to index views

11/26/2015 Copyright G.D. Hager So what does object recognition involve?

11/26/2015 Copyright G.D. Hager Verification: is that a bus?

11/26/2015 Copyright G.D. Hager Detection: are there cars?

11/26/2015 Copyright G.D. Hager Identification: is that a picture of Mao?

11/26/2015 Copyright G.D. Hager Object categorization sky building flag wall banner bus cars bus face street lamp

11/26/2015 Copyright G.D. Hager Scene and context categorization outdoor city traffic …

11/26/2015 Copyright G.D. Hager Challenges 1: view point variation Michelangelo

11/26/2015 Copyright G.D. Hager Challenges 2: illumination slide credit: S. Ullman

11/26/2015 Copyright G.D. Hager Challenges 3: occlusion Magritte, 1957

11/26/2015 Copyright G.D. Hager Challenges 4: scale

11/26/2015 Copyright G.D. Hager Challenges 5: deformation Xu, Beihong 1943

11/26/2015 Copyright G.D. Hager Challenges 6: background clutter Klimt, 1913

11/26/2015 Copyright G.D. Hager History: single object recognition

11/26/2015 Copyright G.D. Hager History: single object recognition Lowe, et al. 1999, 2003 Mahamud and Herbert, 2000 Ferrari, Tuytelaars, and Van Gool, 2004 Rothganger, Lazebnik, and Ponce, 2004 Moreels and Perona, 2005 …

11/26/2015 Copyright G.D. Hager Challenges 7: intra-class variation

11/26/2015 Copyright G.D. Hager History: early object categorization

11/26/2015 Copyright G.D. Hager Turk and Pentland, 1991 Belhumeur et al Schneiderman et al Viola and Jones, 2000 Amit and Geman, 1999 LeCun et al Belongie and Malik, 2002 Schneiderman et al Argawal and Roth, 2002 Poggio et al. 1993

11/26/2015 Copyright G.D. Hager

11/26/2015 Copyright G.D. Hager OBJECTS ANIMALS INANIMATE PLANTS MAN-MADENATURAL VERTEBRATE ….. MAMMALS BIRDS GROUSEBOARTAPIR CAMERA

11/26/2015 Copyright G.D. Hager

11/26/2015 Copyright G.D. Hager Scenes, Objects, and Parts Features Parts Objects Scene E. Sudderth, A. Torralba, W. Freeman, A. Willsky. ICCV 2005.

11/26/2015 Copyright G.D. Hager Object categorization: the statistical viewpoint vs. Bayes rule: posterior ratio likelihood ratioprior ratio

11/26/2015 Copyright G.D. Hager Object categorization: the statistical viewpoint posterior ratio likelihood ratioprior ratio Discriminative methods model posterior Generative methods model likelihood and prior

11/26/2015 Copyright G.D. Hager Discriminative Direct modeling of Zebra Non-zebra Decision boundary

11/26/2015 Copyright G.D. Hager Model and Generative LowMiddle HighMiddle  Low

11/26/2015 Copyright G.D. Hager Three main issues Representation –How to represent an object category Learning –How to form the classifier, given training data Recognition –How the classifier is to be used on novel data

11/26/2015 Copyright G.D. Hager Representation –Generative / discriminative / hybrid

11/26/2015 Copyright G.D. Hager Representation –Generative / discriminative / hybrid –Appearance only or location and appearance

11/26/2015 Copyright G.D. Hager Representation –Generative / discriminative / hybrid –Appearance only or location and appearance –Invariances View point Illumination Occlusion Scale Deformation Clutter etc.

11/26/2015 Copyright G.D. Hager Representation –Generative / discriminative / hybrid –Appearance only or location and appearance –invariances –Part-based or global w/sub-window

11/26/2015 Copyright G.D. Hager Representation –Generative / discriminative / hybrid –Appearance only or location and appearance –invariances –Parts or global w/sub-window –Use set of features or each pixel in image

11/26/2015 Copyright G.D. Hager –Unclear how to model categories, so we learn what distinguishes them rather than manually specify the difference -- hence current interest in machine learning Learning

11/26/2015 Copyright G.D. Hager –Unclear how to model categories, so we learn what distinguishes them rather than manually specify the difference -- hence current interest in machine learning) –Methods of training: generative vs. discriminative Learning

11/26/2015 Copyright G.D. Hager –Unclear how to model categories, so we learn what distinguishes them rather than manually specify the difference -- hence current interest in machine learning) –What are you maximizing? Likelihood (Gen.) or performances on train/validation set (Disc.) –Level of supervision Manual segmentation; bounding box; image labels; noisy labels Learning Contains a motorbike

11/26/2015 Copyright G.D. Hager –Unclear how to model categories, so we learn what distinguishes them rather than manually specify the difference -- hence current interest in machine learning) –What are you maximizing? Likelihood (Gen.) or performances on train/validation set (Disc.) –Level of supervision Manual segmentation; bounding box; image labels; noisy labels –Batch/incremental (on category and image level; user- feedback ) Learning

11/26/2015 Copyright G.D. Hager –Unclear how to model categories, so we learn what distinguishes them rather than manually specify the difference -- hence current interest in machine learning) –What are you maximizing? Likelihood (Gen.) or performances on train/validation set (Disc.) –Level of supervision Manual segmentation; bounding box; image labels; noisy labels –Batch/incremental (on category and image level; user- feedback ) –Training images: Issue of overfitting Negative images for discriminative methods Priors Learning

11/26/2015 Copyright G.D. Hager –Unclear how to model categories, so we learn what distinguishes them rather than manually specify the difference -- hence current interest in machine learning) –What are you maximizing? Likelihood (Gen.) or performances on train/validation set (Disc.) –Level of supervision Manual segmentation; bounding box; image labels; noisy labels –Batch/incremental (on category and image level; user- feedback ) –Training images: Issue of overfitting Negative images for discriminative methods –Priors Learning

11/26/2015 Copyright G.D. Hager –Scale / orientation range to search over –Speed Recognition

11/26/2015 Copyright G.D. Hager State of The Art Current systems deal with simple nearly 2D situations or very restricted input On ~100 categories, reported accuracies in the 70-90% range, but with huge computational loads Not clear we have the right approach (yet)!

11/26/2015 Copyright G.D. Hager Example: Fergus 2003

11/26/2015 Copyright G.D. Hager Results, Fergus 2003

11/26/2015 Copyright G.D. Hager Summary Object recognition/categorization is a rapidly evolving area Current systems are getting to the point they may be useful in real applications. Much more remains to be done in understanding how to move to the next level of performance.