Object Recognition in Images Slides originally created by Bernd Heisele

Fall 2002Pattern Recognition for Vision Object Detection Object Identification Where is Jane? Where is a Face? Is there a face in the image? Who is it? Is it Jane or Erik? Object Recognition

Fall 2002Pattern Recognition for Vision Applicable to many classes of objects Invariance: “External parameters” Pose Illumination “Internal parameters” Person identity Facial expression General Problems of Recognition

Fall 2002Pattern Recognition for Vision Task Given an input image, determine if there are objects of a given class (e.g. faces, people, cars..) in the image and where they are located. Object Detection

Fall 2002Pattern Recognition for Vision Detection—Problems 1.Classifier must generalize over all exemplars of one class. 2.Negative class consists of everything else. 3.High accuracy (small FP rate) required for most applications.

Fall 2002Pattern Recognition for Vision Feature Extraction Search for faces at different resolutions and locations Feature vector (x 1, x 2,…, x n ) Classifier Off-line training Face examples Non-face examples Pixel pattern Classification Result Face Detection – basic scheme

Fall 2002Pattern Recognition for Vision Training Set Train Classifier Labeled Test Set False Positive Correct Classify Sensitivity Training and Testing

Fall 2002Pattern Recognition for Vision Gray Histogram Equalization Masking 19x [0, 1] Histogram Equalization Gradient x-y-Sobel Filtering 17x17 [0, 1] Haar Wavelet Transform 1740 [0, 1] Wavelets Normalization Image Features

Fall 2002Pattern Recognition for Vision ROC Image Features

Fall 2002Pattern Recognition for Vision Classifiers

Fall 2002Pattern Recognition for Vision Real 2900 faces mirrored faces 3D Morphing: Synthetic (T. Vetter, Univ. of Freiburg) Illumination Rotation Identity + = Positive Training Data

Fall 2002Pattern Recognition for Vision Real vs. Synthetic

Fall 2002Pattern Recognition for Vision Problem: 1 face in examined windows Initial set of non- faces Retrain Classifier Add to training set Determine false- positives on large set of non-face images Bootstrapping Negative Training Data

Fall 2002Pattern Recognition for Vision 6.7 FP/image 0.9 FP/image Bootstrapping

Fall 2002Pattern Recognition for Vision Performance of Global Face Detectors

Fall 2002Pattern Recognition for Vision Rotation in the image plane Rotation invariant features Apply 2D rotation to image Rotation out of image plane Component-based classification Train on rotated faces Rotation

Fall 2002Pattern Recognition for Vision Single template Component templates Global vs. Components

Fall 2002Pattern Recognition for Vision Eyes Nose Mouth 1st Level: Components classify 2nd Level: Geometrical relation between components maximum response of each component classifier + x, y location classify Component-based Detection

Fall 2002Pattern Recognition for Vision Synthetic faces: 7 different 3-D head models 2,500 faces Rotation: -30 o to + 30 o 3-D correspondences for automatic location of components Components: discriminatory robust against changes in pose and illumination Learning Components

Fall 2002Pattern Recognition for Vision Start with small initial regions Expand into one of four directions Extract new components from images Train SVM classifiers Choose best expansion according to error bound of SVMs Learning Components—One Way To Do It

Fall 2002Pattern Recognition for Vision x*1x*1 x*2x*2 M R Feature Space M Bound on error E < c(R / M) 2 Margin, Radius and Expected Error Cross Validation might be better

Fall 2002Pattern Recognition for Vision Some Examples

Fall 2002Pattern Recognition for Vision About 40,000 faces 68 people 13 poses 43 illumination conditions 4 different expressions Faces have been manually labeled (only –45 o to 45 o of rotation) Test on CMU PIE Database

Fall 2002Pattern Recognition for Vision Heisele, B., T. Serre, M. Pontil, T. Vetter and T. Poggio. Categorization by Learning and Combining Object Parts. In: Advances in Neural Information Processing Systems (NIPS'01), Vancouver, Canada ROC Component vs. Global

Fall 2002Pattern Recognition for Vision (From C. Papageorgiou, 2000) Representation: dictionary of Haar wavelets; high dimensional feature space (>1300 features) SVM classifier Pedestrian Detection

Fall 2002Pattern Recognition for Vision Examples

Fall 2002Pattern Recognition for Vision Components Haar wavelets 5 components Can deal with partial occlusions (From A. Mohan, 1999)

Fall 2002Pattern Recognition for Vision Components are small, and prone to false detection, even within the face. Advances on Component-base Face Detection Stan Bileschi Stan Bileschi

Fall 2002Pattern Recognition for Vision Use the remainder of the face in the negative training set Positive Negative Training on Faces

Fall 2002Pattern Recognition for Vision Red: Trained only with faces. Training on Faces Only Blue: Trained on faces and non- faces.

Fall 2002Pattern Recognition for Vision Errors Often, many components classify correctly, with only a few errors

Fall 2002Pattern Recognition for Vision Using Models of Pair-wise Positions

Fall 2002Pattern Recognition for Vision Pair-wise Biasing Leads to Tightened Result Images

Fall 2002Pattern Recognition for Vision Pair-wise Biasing

Fall 2002Pattern Recognition for Vision Application: Eye Detection

Fall 2002Pattern Recognition for Vision Task: Given an image of an object of a particular class (e.g. face) identify which exemplar it is. B B D D C A C A Identification

Fall 2002Pattern Recognition for Vision Identification—Problems 1.Multi-class problem 2.Classifier must distinguish between exemplars that might look very similar. 3.Classifier has to reject exemplars that were not in the training database.

Fall 2002Pattern Recognition for Vision Limited information in a single face image Illumination Rotation Problems in Face Identification

Fall 2002Pattern Recognition for Vision Face Image Gray, Gradient, Wavelets, … Feature extraction Support Vector Machine, …. Classifier Training Data A Identification Result System Architecture

Fall 2002Pattern Recognition for Vision Multi-class Classification with SVM A Bottom-Up 1vs1 A or B or C or D CDB A or B C or D Training: L (L-1) / 2 Classification : L-1 A BA,C,D CA,B,D DA,B,C 1 vs. All B,C,D Training: L Classification : L

Fall 2002Pattern Recognition for Vision Global Approach Detect and extract face Feed gray values into N SVMs Classify based on maximum output SVM A SVM B SVM C SVM D Max Operation Identification result

Fall 2002Pattern Recognition for Vision Global Approach with Clustering SVM C SVM C SVM C SVM C Max Operation Identification result Partition training images of each person into viewpoint- specific clusters Train a linear SVM on each cluster Take maximum over all SVM outputs

Fall 2002Pattern Recognition for Vision Component-based Approach Detect and extract components Feed gray values of components to N SVMs Take max. over all SVM outputs SVM A SVM B SVM C SVM D Max Operation Identification Results

Fall 2002Pattern Recognition for Vision Component-based Face Recognition with 3D Morphable Models Feature 1 != Feature 2 Feature 1 = Feature 2 Why Components for Identification? Jennifer Huang

Fall 2002Pattern Recognition for Vision Heisele, B., P. Ho and T. Poggio. Face Recognition with Support Vector Machines: Global Versus Component-based Approach, International Conference on Computer Vision (ICCV'01), Vancouver, Canada, Vol. 2, , More ROC Curves

Fall 2002Pattern Recognition for Vision 3D morphable models A Morphable Model for the Synthesis of 3D Faces. Blanz, V. and Vetter, T. SIGGRAPH'99 Conference Proceedings, pp Training data for component-based face recognition Morphable Models for Face Identification, Jennifer Huang Jennifer Huang

Fall 2002Pattern Recognition for Vision Generation of 3D head model from two images See class on Morphable Models Morphable Model Jennifer Huang

Fall 2002Pattern Recognition for Vision Synthetic images are easily rendered from 3D head model under varying illuminations and rotations in depth Some Training Images Jennifer Huang

Fall 2002Pattern Recognition for Vision Preliminary Results on Synthetic Images Jennifer Huang

Fall 2002Pattern Recognition for Vision Problems Encountered: Detection Inaccurate Component detection Recognition Accuracy of 3D models Choice of Illumination and Pose Current Work – Testing on Real Images Jennifer Huang

Fall 2002Pattern Recognition for Vision Literature B. Heisele, A. Verri and T. Poggio: Learning and Vision Machines. Proceedings of the IEEE, Visual Perception: Technology and Tools, Vol. 90, No. 7, pp , See also CBCL Web page