1. Content-Based Image Retrieval
Jeff Bigham
So far we haven’t really touched upon what you’d think would be one of the fundamental goals of computer vision. And that is, recognizing objects within pictures or videos well enough to allow one to search for that object by name and have pictures of it returned to you. Much like, you type words into Google and it returns documents that contain those words.

2. Who is Steve Seitz? 1. 7. 4.
10. 2. 8.
Top 10 results from Google Images.
11. 5. 3. 9. 6.
12.

3. Goals Efficient use of Visual Resources Effective Search
Intuitive Interfaces
Understanding?

4. Image Search Landscape
Inherits problems of text search
Ambiguity
Spam
But vision is hard
uses little or no “vision”
often reduces to text search
Rich UI
Text
Images
Video

5. Find me an image of a…
“Pretty girl doing something active, sporty in a summery setting, beach - not wearing lycra, exercise clothes - more relaxed in tee-shirt. Feature is about deodorant so girl should look active - not sweaty but happy, healthy, carefree - nothing too posed or set up - nice and natural looking.”
A typical query issued to a manual image search by a marketing person that illustrates why CB-IR is very hard.

6. images.google.com & flickr.com
Collect Associated Text
alt, title, longdesc attributes of img tag
Information from path (e.g., cow.gif)
nearby text on page
Refinement
remove duplicates
cluster-based improvement
Other
PageRank, etc.

7. Matching Words and Pictures
Learn text/image segment association
Segment image
Generate feature vector for each segment
Associate text with the segment features
Build generative probabilistic model for text given segmentation features

8. Matching Words and Pictures
Kobus Barnard, Pinar Duygulu, Nando de Freitas, David Forsyth, David Blei, and Michael I. Jordan, "Matching Words and Pictures", Journal of Machine Learning Research, Vol 3, pp , 2003.

9. Other Ways of Searching
Search by Text
Search by Example
Search by Drawing
Search by Model

10. Other Search Interfaces?
Josef Sivic, Andrew Zisserman, "Video Google: A Text Retrieval Approach to Object Matching in Videos," iccv, p. 1470,  Ninth IEEE International Conference on Computer Vision (ICCV'03) - Volume 2,  2003.

11. Video Google
Viewpoint Invariant Descriptors

12. Video Google
Visual Vocabulary

13. Inverted Index Document 1 Now is the time for all good men
Word
Document
aid 1
all
and 2
can
come
1, 2
country
for
good …
the
1, 1, 2
Document 1
Now is the time
for all good men
to come to the aid
of their country.
Document 2
Summer has come
and passed. The innocent
can never last.

14. Video Google
(Demo of Video Google)

15. Computer Vision is Hard
But, humans do it well
Luis von Ahn
(ESP Game Demo)

16. Vision as Security?
True image search is a hard AI problem

17. The End