Unsupervised Learning of Categories from Sets of Partially Matching Image Features Kristen Grauman and Trevor Darrel CVPR 2006 Presented By Sovan Biswas MSc (Engg), SERC, IISc

Goal Automatically recover categories from an unlabeled collection of images, and form predictive classifiers to label new images Tolerate clutter, occlusion, common transformations Allow optional, variable amount of supervision Efficiency

Motivation Majority of the object recognition and categorization requires ‘Manual Annotations’. ‘Manual Annotations’ : practical limit on number of classes and number of training examples per class. Introduces Biasing : effects the performance. Unsupervised recognition and categorization would be cost effective and reduce burden.

Motivation Majority of the object recognition and categorization requires ‘Manual Annotations’. ‘Manual Annotations’ : – practical limit on number of classes and number of training examples per class. Introduces Biasing : – effects the performance. Unsupervised recognition and categorization would be cost effective and reduce burden.

Related work Modeling scenes with local descriptors ICCV 2005, P. Quelhas, F. Monay, etc. Discovering Object Categories in Image Collections ICCV 2005, J. Sivic, B. Russell, A. Efros, etc. Learning Object Categories from Google’s Image Search ICCV 2005, R. Fergus, L. Fei-Fei, etc.

Related work (contd.) Local features – recognition and retrieval Unordered feature vectors and Vary in size Previous Approaches : – vector quantization – Build codebook of feature descriptors – Apply conventional clustering methods or latent semantic analysis (LSA) Issues : – Clutters – Occlusions – Addition of small amount of labeled data.

Approach Pair wise affinities reflecting partial-match feature correspondences are computed for all input images. (Optional) A variable amount of supervised labels (pairing constraints) are optionally collected from the user and affinity matrix are adjusted accordingly. Spectral clustering is used to recover the initial dominant clusters. Prototypical examples : by evaluating the typical “feature masks” contributing to each within-cluster partial matching Top-ranked prototypical examples from refined groupings compose the learned categories, which are used to train a set of predictive classifiers for labelling unseen examples.

Pyramid match kernel Number of newly matched pairs at level i Measure of difficulty of a match at level i Approximate partial match similarity

Sets of features

Feature extraction, Histogram pyramid: level i has bins of size 2 i

Pyramid match kernel Weights inversely proportional to bin size Normalize kernel values to avoid favoring large sets measure of difficulty of a match at level i histogram pyramids number of newly matched pairs at level i

Pyramid match kernel

Algorithm 1) Initial Grouping Feature Sets with partial correspondences. 2) Inferring Category Feature Masks. 3) Identifying Prototypes.

Initial Grouping Data Set U = {I 1, I 2, …, I N } – where I i are images. I i is denoted by feature set – X i = {f 1, …, f mi } where m i may vary across U – f i : description vectors. Authors uses SHIFT for local feature.

Initial Grouping (contd.) Derived Partial Correspondence using Pyramid Match Kernel.

Initial Grouping (contd.) Normalized cuts criterion to create preliminarily classes. Optionally Introduce of weak semi supervision in the form of pairwise constraints between the unlabeled images Adjust the weight of affinity matrix to highest value for “must group” and zero to “cannot – group”.

Initial Grouping (contd.) Background feature matches Multiple object matches

Inferring Category Feature Masks Confounding normalized cuts. Goal : Identify prototypical example. Intuition : Inlier uses similar partial portion of feature sets to form partial matches. Use Modified pyramid Kernel.

Identifying Prototypes Use the feature mask to re weight the individual feature. Sort the images based on how consistent they match with the remainder of sets. Examples within top threshold percentile are “prototype candidates” Optionally allow user constraints and adjust affinity matrix in case of disagreement

Inferring feature masks feature index contribution to match

Inferring feature masks weighted feature mask

Refining intra-cluster matches weighted feature mask

Refining intra-cluster matches weighted feature mask

Refining intra-cluster matches weighted feature mask

Refining intra-cluster matches weighted feature mask

Refining intra-cluster matches weighted feature mask

Selecting category prototypes

Unsupervised recovery of category prototypes Top percentile of prototypes Prototype accuracy / category Caltech-4 data set

Semi-supervised category labeling Caltech-4 data set Recognition rate / class Amount of supervisory information (number of “must-group” pairs)

Future Work Number of groupings are needed to specified during normalized cuts. (limitations) Finding of most critical local features and interest points. (performance) Using the algorithm for image retrival.

References Shape Matching and Object Recognition Using Shape Contexts S. Belongie, J. Malik, and J. Puzicha, IEEE Tranc. PAMI 2002 Shape Matching and Object Recognition using Low Distortion Correspondences A. Berg, T. Berg, and J. Malik, IEEE CVPR 2005 Distinctive Image Features from Scale Invariant Key points D Lowe, International Journal of Computer Vision 2004 Indexing Based on Scale Invariant Interest Points K. Mikolajczyk and C. Schmid, International Journal of Computer Vision 2004 The Pyramid Match Kernel: Discriminative Classification with Sets of Image Features K. Grauman and T. Darrell, IEEE ICCV 2005 Normalized Cuts and Image Segmentation J. Shi and J. Malik, IEEE Tranc. PAMI 2000