The topic discovery models Discovering Objects and their Location in Images Josef Sivic1, Bryan C. Russell2, Alexei A. Efros3, Andrew Zisserman1 and William T. Freeman2 MIT CMU 1Oxford University 2MIT 3Carnegie Mellon University CMU MIT Introduction The topic discovery models Improving localization using doublets Goal: Discover visual object categories and their segmentation given a collection of unlabelled images Probabilistic Latent Semantic Analysis (pLSA) [Hofmann’99] Form a new vocabulary from pairs of locally co-occurring regions w … visual words d … documents (images) z … topics (‘objects’) Doublet formation Doublet example I Doublet examle II pLSA graphical model P(z|d) and P(w|z) are multinomial distributions pLSA Model fitting: Find topic vectors P(w|z) common to all documents and mixture coefficients P(z|d) specific to each document. Fit model by maximizing likelihood of data using EM. All detected visual words Singlet segmentation Singlet segmentation Doublet segmentation Bounding box overlap score for faces, singlets: 0.49, doublets: 0.61 Approach: 1) Represent an image as a collection of visual words 2) Apply topic discovery models from statistical text analysis Latent Dirichlet Allocation (LDA) [Blei et al.’03] (see paper for more details) Treat multinomial weights over topics as random variables. Fit model using Gibbs sampling [Griffiths and Steyvers’04]. Experiment II: MIT dataset 2873 images, learn 10 topics Image representation 4 of the 10 learned topics shown by the 5 most probable images for each topic Represent an image as a histogram of “visual words” LDA graphical model 2 1 ... Results “Buildings” “Trees / Grass” Results shown only for pLSA. LDA had very similar performance. Experiment I: Caltech Dataset Histogram of visual words Four object categories: faces, motorbikes, airplanes and cars rear (total of 3,190 images) and 900 background images “Computers” “Bookshelves” Detect affine covariant regions Represent each region by a SIFT descriptor Build visual vocabulary by k-means clustering (K~1,000) Assign each region to the nearest cluster centre Mikolajczyk and Schmid’02, Schaffalitzky and Zisserman’02, Matas et al. ’02, Lowe’99, Sivic and Zisserman’03 Image Classification Example Images with multiple objects Assign each image to a topic with the highest P(z|d) Learn K = (5,6,7) topics Background is better modelled by multiple topics Pre-learning background topics on a separate bg dataset improves results Performance on novel images is comparable with weakly supervised method of [Fergus et al.’03] Examples of visual words Confusion tables (K=5,6,7) learned topics Experiment III: Application to image retrieval Segmentation Five samples from a ‘motorbike’ visual word Visual Polysemy. Single visual word occurring on different (but locally similar) parts on different object categories. Learn topic vectors on Caltech database Represent new query image in terms of learned topic vectors For a given word wi in document dj examine posterior probability over topics. pLSA Retrieve images within Caltech database Visual words colour coded according to the topic with the highest probability Raw word histograms Five samples from an ‘airplane’ visual word Visual Synonyms. Two different visual words representing a similar part of an object (wheel of a motorbike). Faces Motorbikes Query image Airplanes Cars Precision – Recall plot Overview Retrieve images in movie Pretty Woman Background I Background II Retrieved images using pLSA ‘object’ coefficients P(z|d) Find visual words Form histograms Background III Example face segmentation Pretty Woman (6,641 keyframes) Retrieved images using visual word histograms Represent each keyframe using topic vectors learned on Caltech database Discover topics Example motorbike segmentation Example airplane segmentation