1. Agenda Introduction Bag-of-words models Visual words with spatial location Part-based models Discriminative methods Segmentation and recognition Recognition-based image retrieval Datasets & Conclusions

2. Retrieval domains Internet image search Video search for people/objects Searching home photo collections

3. Learning from Internet Image Search Joint learning of text and images Large scale retrieval

5. Noisy labels

6. Improving Google’s Image Search Fergus, Fei-Fei, Perona, Zisserman, ICCV 2005 Variant of pLSA that includes spatial information

7. Topics in model Re-ranking result: Motorbike Automatically chosen topic

8. Animals on the Web Berg and Forsyth, CVPR 2006 Gather images using text search Use LDA to discover “good” images using features based on nearby text, shape, color

9. Boostrapping of Image Search 2 4 Images returned with PENGUIN query Removal of drawings and abstract images Naives Bayes ranking using noisy metadata Train SVM……. Schroff, Zisserman, Criminisi, Harvesting Image Databases from the Web, ICCV 2007 Final ranking using SVM

10. OPTIMOL Li, Wang, Fei-Fei CVPR 07

11. Learning from Internet Image Search Joint learning of text and images Large scale retrieval

12. Matching Words and Pictures Barnard, Duygulu, de Freitas, Forsyth, Blei, Jordan, JMLR 2003

13. Text to Images

14. Images to text Use Blobworld or nCuts to segments images into regions Need to deduce labels attached to each image

15. Images to text result

16. Names and Faces in the News Berg, Berg, Edwards, Maire, White, Teh, Learned-Miller, Forsyth. CVPR 2004 1.Find faces (standard face detector), rectify them to same pose. 2.Perform Kernel PCA and Linear Discriminant Analysis (LDA). 3.Extract names from text. 4.Cluster faces, with each name corresponding to a cluster. 5.Use language model to refine results Collected 500,000 images and text captions from Yahoo! News

17. Initial clusters

18. Clusters refined with language model

19. Learning from Internet Image Search Joint learning of text and images Large scale retrieval

20. Vocabulary tree Nistér & Stewénius CVPR 2006. KD-tree in descriptor space Inverse lookup of features Specific object recognition  Not category-level

21. Slide from D. Nister

40. Pyramid Match Hashing Grauman & Darell, CVPR 2007 Combines Pyramid Match Kernel (efficient computation of correspondences between two set of vectors) with Locality Sensitive Hashing (LSH) [Indyk & Motwani 98] Allows matching of the set of features in a query image to sets of features in other images in time that is sublinear in # images Theoretical guarantees

41. Salakhutdinov and Hinton, SIGIR 2007 Torralba, Fergus, Weiss, CVPR 2008 Map images to compact binary codes Hash codes for fast lookup Semantic Hashing