Content-Based Image Retrieval Jeff Bigham So far we haven’t really touched upon what you’d think would be one of the fundamental goals of computer vision. And that is, recognizing objects within pictures or videos well enough to allow one to search for that object by name and have pictures of it returned to you. Much like, you type words into Google and it returns documents that contain those words.
Who is Steve Seitz? 1. 7. 4. 10. 2. 8. Top 10 results from Google Images. 11. 5. 3. 9. 6. 12.
Goals Efficient use of Visual Resources Effective Search Intuitive Interfaces Understanding?
Image Search Landscape Inherits problems of text search Ambiguity Spam But vision is hard uses little or no “vision” often reduces to text search Rich UI Text Images Video
Find me an image of a… “Pretty girl doing something active, sporty in a summery setting, beach - not wearing lycra, exercise clothes - more relaxed in tee-shirt. Feature is about deodorant so girl should look active - not sweaty but happy, healthy, carefree - nothing too posed or set up - nice and natural looking.” A typical query issued to a manual image search by a marketing person that illustrates why CB-IR is very hard. http://images.google.com
images.google.com & flickr.com Collect Associated Text alt, title, longdesc attributes of img tag Information from path (e.g., cow.gif) nearby text on page Refinement remove duplicates cluster-based improvement Other PageRank, etc.
Matching Words and Pictures Learn text/image segment association Segment image Generate feature vector for each segment Associate text with the segment features Build generative probabilistic model for text given segmentation features
Matching Words and Pictures Kobus Barnard, Pinar Duygulu, Nando de Freitas, David Forsyth, David Blei, and Michael I. Jordan, "Matching Words and Pictures", Journal of Machine Learning Research, Vol 3, pp 1107-1135, 2003.
Other Ways of Searching Search by Text Search by Example Search by Drawing Search by Model
Other Search Interfaces? Josef Sivic, Andrew Zisserman, "Video Google: A Text Retrieval Approach to Object Matching in Videos," iccv, p. 1470, Ninth IEEE International Conference on Computer Vision (ICCV'03) - Volume 2, 2003.
Video Google Viewpoint Invariant Descriptors
Video Google Visual Vocabulary
Inverted Index Document 1 Now is the time for all good men Word Document aid 1 all and 2 can come 1, 2 country for good … the 1, 1, 2 Document 1 Now is the time for all good men to come to the aid of their country. Document 2 Summer has come and passed. The innocent can never last.
Video Google (Demo of Video Google)
Computer Vision is Hard But, humans do it well Luis von Ahn (ESP Game Demo)
Vision as Security? True image search is a hard AI problem
The End