By Suren Manvelyan, http://www.surenmanvelyan.com/gallery/7116 Crocodile (nile crocodile?) By Suren Manvelyan, http://www.surenmanvelyan.com/gallery/7116

By Suren Manvelyan, http://www.surenmanvelyan.com/gallery/7116 horse By Suren Manvelyan, http://www.surenmanvelyan.com/gallery/7116

By Suren Manvelyan, http://www.surenmanvelyan.com/gallery/7116 fish By Suren Manvelyan, http://www.surenmanvelyan.com/gallery/7116

By Suren Manvelyan, http://www.surenmanvelyan.com/gallery/7116 iguana By Suren Manvelyan, http://www.surenmanvelyan.com/gallery/7116

By Suren Manvelyan, http://www.surenmanvelyan.com/gallery/7116 snail By Suren Manvelyan, http://www.surenmanvelyan.com/gallery/7116

By Suren Manvelyan, http://www.surenmanvelyan.com/gallery/7116 rabbit By Suren Manvelyan, http://www.surenmanvelyan.com/gallery/7116

By Suren Manvelyan, http://www.surenmanvelyan.com/gallery/7116 Tiger python By Suren Manvelyan, http://www.surenmanvelyan.com/gallery/7116

By Suren Manvelyan, http://www.surenmanvelyan.com/gallery/7116 Siam cat By Suren Manvelyan, http://www.surenmanvelyan.com/gallery/7116

By Suren Manvelyan, http://www.surenmanvelyan.com/gallery/7116 Husky with heterochromia By Suren Manvelyan, http://www.surenmanvelyan.com/gallery/7116

Indy http://www.surenmanvelyan.com/gallery/7116 Crocodile (nile crocodile?) Indy

Indy http://www.surenmanvelyan.com/gallery/7116 Crocodile (nile crocodile?) Indy

Image Categorization Training Testing Training Labels Training Images Image Features Classifier Training Trained Classifier Testing Image Features Trained Classifier Prediction Outdoor Test Image

History of ideas in recognition 1960s – early 1990s: the geometric era 1990s: appearance-based models Mid-1990s: sliding window approaches Late 1990s: local features Early 2000s: parts-and-shape models Mid-2000s: bags of features Svetlana Lazebnik

Bag-of-features models Svetlana Lazebnik

Bag-of-features models Bag of ‘words’ Object Svetlana Lazebnik

Objects as texture All of these are treated as being the same No distinction between foreground and background: scene recognition? Motivation slide. Clearly location has to help, but bag-of-words doesn’t have it… Svetlana Lazebnik

Origin 1: Texture recognition Texture is characterized by the repetition of basic elements or textons For stochastic textures, it is the identity of the textons, not their spatial arrangement, that matters Julesz, 1981; Cula & Dana, 2001; Leung & Malik 2001; Mori, Belongie & Malik, 2001; Schmid 2001; Varma & Zisserman, 2002, 2003; Lazebnik, Schmid & Ponce, 2003

Origin 1: Texture recognition histogram Universal texton dictionary Julesz, 1981; Cula & Dana, 2001; Leung & Malik 2001; Mori, Belongie & Malik, 2001; Schmid 2001; Varma & Zisserman, 2002, 2003; Lazebnik, Schmid & Ponce, 2003

Origin 2: Bag-of-words models Orderless document representation: frequencies of words from a dictionary Salton & McGill (1983)

Origin 2: Bag-of-words models Orderless document representation: frequencies of words from a dictionary Salton & McGill (1983) US Presidential Speeches Tag Cloud http://chir.ag/phernalia/preztags/

Origin 2: Bag-of-words models Orderless document representation: frequencies of words from a dictionary Salton & McGill (1983) US Presidential Speeches Tag Cloud http://chir.ag/phernalia/preztags/

Origin 2: Bag-of-words models Orderless document representation: frequencies of words from a dictionary Salton & McGill (1983) US Presidential Speeches Tag Cloud http://chir.ag/phernalia/preztags/

Bag-of-features steps Extract features Learn “visual vocabulary” Quantize features using visual vocabulary Represent images by frequencies of “visual words”

1. Feature extraction Regular grid or interest regions

1. Feature extraction Detect patches Compute descriptor Normalize patch Detect patches Slide credit: Josef Sivic

1. Feature extraction … Slide credit: Josef Sivic

2. Learning the visual vocabulary … Slide credit: Josef Sivic

2. Learning the visual vocabulary … Clustering Slide credit: Josef Sivic

2. Learning the visual vocabulary … Clustering Slide credit: Josef Sivic

K-means clustering Want to minimize sum of squared Euclidean distances between points xi and their nearest cluster centers mk Algorithm: Randomly initialize K cluster centers Iterate until convergence: Assign each data point to the nearest center Recompute each cluster center as the mean of all points assigned to it

Clustering and vector quantization Clustering is a common method for learning a visual vocabulary or codebook Unsupervised learning process Each cluster center produced by k-means becomes a codevector Codebook can be learned on separate training set Provided the training set is sufficiently representative, the codebook will be “universal” The codebook is used for quantizing features A vector quantizer takes a feature vector and maps it to the index of the nearest codevector in a codebook Codebook = visual vocabulary Codevector = visual word

Example codebook … Appearance codebook Source: B. Leibe

Visual vocabularies: Issues How to choose vocabulary size? Too small: visual words not representative of all patches Too large: quantization artifacts, overfitting Computational efficiency Vocabulary trees (Nister & Stewenius, 2006)

But what about layout? All of these images have the same color histogram

Spatial pyramid Compute histogram in each spatial bin

Spatial pyramid representation Extension of a bag of features Locally orderless representation at several levels of resolution level 0 Lazebnik, Schmid & Ponce (CVPR 2006)

Spatial pyramid representation Extension of a bag of features Locally orderless representation at several levels of resolution level 1 level 0 Lazebnik, Schmid & Ponce (CVPR 2006)

Spatial pyramid representation Extension of a bag of features Locally orderless representation at several levels of resolution level 2 level 0 level 1 Lazebnik, Schmid & Ponce (CVPR 2006)

Scene category dataset Multi-class classification results (100 training images per class)

Caltech101 dataset http://www.vision.caltech.edu/Image_Datasets/Caltech101/Caltech101.html Multi-class classification results (30 training images per class)

Bags of features for action recognition Space-time interest points Juan Carlos Niebles, Hongcheng Wang and Li Fei-Fei, Unsupervised Learning of Human Action Categories Using Spatial-Temporal Words, IJCV 2008.

History of ideas in recognition 1960s – early 1990s: the geometric era 1990s: appearance-based models Mid-1990s: sliding window approaches Late 1990s: local features Early 2000s: parts-and-shape models Mid-2000s: bags of features Present trends: combination of local and global methods, context, deep learning Svetlana Lazebnik

Large-scale Instance Retrieval Computer Vision James Hays Many slides from Derek Hoiem and Kristen Grauman

? Multi-view matching vs … Matching two given views for depth Search for a matching view for recognition Kristen Grauman

Video Google System Collect all words within query region Inverted file index to find relevant frames Compare word counts Spatial verification Sivic & Zisserman, ICCV 2003 Demo online at : http://www.robots.ox.ac.uk/~vgg/research/vgoogle/index.html Retrieved frames Kristen Grauman 48

Application: Large-Scale Retrieval Query Results from 5k Flickr images (demo available for 100k set) [Philbin CVPR’07]

http://www.google.com/mobile/goggles

Simple idea See how many keypoints are close to keypoints in each other image Lots of Matches Few or No Matches But this will be really, really slow!

Indexing local features Each patch / region has a descriptor, which is a point in some high-dimensional feature space (e.g., SIFT) Descriptor’s feature space Kristen Grauman 55

Indexing local features When we see close points in feature space, we have similar descriptors, which indicates similar local content. Query image Descriptor’s feature space Database images Easily can have millions of features to search! Kristen Grauman

Indexing local features: inverted file index For text documents, an efficient way to find all pages on which a word occurs is to use an index… We want to find all images in which a feature occurs. To use this idea, we’ll need to map our features to “visual words”. Kristen Grauman

Descriptor’s feature space Visual words Map high-dimensional descriptors to tokens/words by quantizing the feature space Quantize via clustering, let cluster centers be the prototype “words” Word #2 Determine which word to assign to each new image region by finding the closest cluster center. Descriptor’s feature space Kristen Grauman

Visual words Example: each group of patches belongs to the same visual word Figure from Sivic & Zisserman, ICCV 2003 Kristen Grauman

Visual vocabulary formation Issues: Vocabulary size, number of words Sampling strategy: where to extract features? Clustering / quantization algorithm Unsupervised vs. supervised What corpus provides features (universal vocabulary?) Kristen Grauman

Sampling strategies Sparse, at interest points Dense, uniformly Randomly To find specific, textured objects, sparse sampling from interest points often more reliable. Multiple complementary interest operators offer more image coverage. For object categorization, dense sampling offers better coverage. [See Nowak, Jurie & Triggs, ECCV 2006] Multiple interest operators K. Grauman, B. Leibe Image credits: F-F. Li, E. Nowak, J. Sivic

Inverted file index Database images are loaded into the index mapping words to image numbers Kristen Grauman

Inverted file index New query image is mapped to indices of database images that share a word. Kristen Grauman

Inverted file index Key requirement for inverted file index to be efficient: sparsity If most pages/images contain most words then you’re no better off than exhaustive search. Exhaustive search would mean comparing the word distribution of a query versus every page.

Instance recognition: remaining issues How to summarize the content of an entire image? And gauge overall similarity? How large should the vocabulary be? How to perform quantization efficiently? Is having the same set of visual words enough to identify the object/scene? How to verify spatial agreement? How to score the retrieval results? Kristen Grauman