1. Computer Vision, Part 1

6. Topics for Vision Lectures 1.Content-Based Image Retrieval (CBIR) 2.Object recognition and scene understanding

7. Content-Based Image Retrieval

8. Example: Google Search by ImageGoogle Search by Image

9. Query Image Extract Features (Primitives) Image Database Features Database Similarity Measure Matched Results Relevance Feedback Algorithm From http://www.amrita.edu/cde/downloads/ACBIR.ppt Basic technique Each image in database is represented by a feature vector: x 1, x 2,...x N, where x i = (x i 1, x i 2, …, x i m ) Query is represented in terms of same features: Q =(Q 1, Q 2, …, Q m ) Goal: Find stored image with vector x i most similar to query vector Q

10. Distance measure: Possible distance measures d(Q, x i ): – Inner (dot) product – Histogram distance (for histogram features) – Graph matching (for shape features).

11. Some issues in designing a CBIR system Query format, ease of querying Speed Crawling, preprocessing Interactivity, user relevance feedback Visual features which to use? How to combine? Curse of dimensionality Indexing Evaluation of performance

12. Types of Features Typically Used Intensities Color Texture Shape Layout

13. Intensity histograms http://www.clear.rice.edu/elec301/Projects02/ artSpy/intensity.html

14. Color Features Hue, saturation, value

15. http://www.owlnet.rice.edu/~elec301/Projects02/artSpy/patmac/mcolhist.gif Color Histograms (8 colors)

16. http://www.owlnet.rice.edu/~elec301/Projects02/artSpy/patmac/mcolhist.gif

17. Color auto-correlogram Pick any pixel p 1 of color C i in the image I. At distance k away from p 1 pick another pixel p 2. What is the probability that p 2 is also of color C i ? p1p1 p2p2 k Red ? Image: I From: http://www.cse.ucsc.edu/classes/ee264/Winter02/xgfeng.ppt

18. The auto-correlogram of image I for color C i, distance k: Integrates both color information and space information. From: http://www.cse.ucsc.edu/classes/ee264/Winter02/xgfeng.ppt

19. Two images with their autocorrelograms. Note that the change in spatial layout would be ignored by color histograms, but causes a significant difference in the autocorrelograms. From http://www.cs.cornell.edu/rdz/Papers/ecdl2/spatial.htm

20. Color histogram rank: 411; Auto-correlogram rank: 1 Color histogram rank: 310; Auto-correlogram rank: 5 Color histogram rank: 367; Auto-correlogram rank: 1 From: http://www.cs.cornell.ed u/rdz/Papers/ecdl2/spati al.htm

21. Texture representations Gray-level co-occurrence Entropy Contrast Fourier and wavelet transforms Gabor filters

22. Texture Representations Each image has the same intensity distribution, but different textures Can use auto-correlogram based on intensity (gray-level co-occurrence)

23. Texture from entropy Images filtered by entropy: Each output pixel contains entropy value of 9x9 neighborhood around original pixel From: http://www.siim2011.org/abstracts/advanced_visualization_tools_ss_pao.html

24. Texture from fractal dimension From: http://www.cs.washington.edu/homes/rahul/data/ic cv07.pdf

25. Texture from Contrast Example : http://www.clear.rice.edu/elec301/Projects02/artSpy/grainin ess.html http://www.clear.rice.edu/elec301/Projects02/artSpy/grainin ess.html

26. Texture from Wavelets http://www.clear.rice.edu/elec301/Projects02/ artSpy/dwt.html

28. Shape representations Some of these need segmentation (another whole story!) Area, eccentricity, major axis orientation Skeletons, shock graphs Fourier transformation of boundary Histograms of edge orientations

29. From: http://www.lems.brown.edu/vision/researchAreas/ShockMatching/shock-ed-match-results1.gif

30. From: http://www.cs.ucl.ac.uk/staff/k.jacobs/teaching/prmv/Edge_histogramming.jpg Histogram of edge orientations

31. Visual abilities largely missing from current CBIR systems Object recognition Perceptual organization Similarity between semantic concepts

32. Image 1Image 2 Examples of semantic similarity

33. Image 1Image 2 Examples of semantic similarity

34. Image 1 Image 2 Examples of semantic similarity

35. In general, current systems have not yet had significant impact on society due to an inability to bridge the semantic gap between computers and humans.

37. Image Understanding and Analogy- Making

38. Bongard problems as an idealized domain for exploring the semantic gap