1. Survey of Algorithms to Query Image Databases COMP 290-72:Computational Geometry Benjamin Lok 11/2/98 Image from Kodak’s PhotoQuilt

2. Outline of Talk w Overview of the problem w Three methods Color based Shape based Vision based w Conclusions Image from Microsoft Clip Gallery

3. Problem w Query an image database w What does a “match” mean? Application dependent Notion of subjectivity Sensitivity to noise

4. Problem w Semantic similarity is still not possible ex. “All images with cats” w To determine similarity, we need to a new: Metric Space Images from Microsoft Clip Gallery and website

5. Ambiguity Guy Girl Guy Images from “Shrine to Long Haired Men” and “Videos of Women Getting Their Heads Shaved” websites

6. Yossi Rubner, Leonidas Guibas, Carlos Tomasi (1997) Stanford Vision Laboratory The Earth Mover’s Distance, Multi-Dimensional Scaling, and Color-Based Image Retrieval Image from Microsoft Clip Gallery

7. Color Signatures w Utilize the CIE-LAB color space Based on human perception of color w Map each pixel to a point in color space Common color values increase weight of point w Group clusters into points (8-12 per image) Rubner, Guibas, and Tomasi

8. Earth Mover’s Distance w To compare two images, compute the “work” needed to move the cluster points from one image to the other Rubner, Guibas, and Tomasi

9. Earth Mover’s Distance (cont) w Solving a linear programming problem: Given two signatures: p = {(p 1,w p1 ),…,(p m,w pm )} and q = {(q 1,w q1 ),…,(q n,w qn )} Find C where C ij is the amount of weight p i matched q j

10. Applications w Visualize Databases (Queries and Results) w Scale the multiple dimensions into 2D using MDS and minimize STRESS Rubner, Guibas, and Tomasi

11. Database Visualization Rubner, Guibas, and Tomasi

12. Algorithm Recap w Map pixels to 3D color space points w Locate and compress “clusters” of points w 8 to 12 points determine the color signature w Calculate the Earth Mover’s Distance to determine “distance” between two images Image from YenPen Stationary Website

13. Advantages w Based on human perception of color w Some invariance to small change in viewpoint and lighting w Meaningful metric w Relatively fast w Can embed multiple metrics Disadvantages w Depending on application, query format might be not be intuitive w Not much use for non- color images w False positives a real possibility depending on working domain

14. Shaped-based Image Retrieval Using Geometric Hashing Scott D. Cohen and Leonidas J. Guibas 1997 Stanford Vision Laboratory Image from Microsoft Clip Gallery

15. Overview w Implementation Search through 500 Chinese characters w Goals Provide invariance to scale, rotation, and translation speed and accuracy Cohen and Guibas

16. Generating a Illustration w Illustration - set of curves that summarize an image Edgel detection Medial Axis determination Cohen and Guibas

17. Approximating with Polylines w Convert medial axis representation to polylines w Tradeoff between speed and accuracy Cohen and Guibas

18. Geometric Hashing w Geometric Hashing - method used to compare two point sets under some transformation group w Take each point and use it as the origin of a coordinate system Cohen and Guibas

19. Geometric Hashing (cont) w If translating P by q j - p i produces a good match I i (P) and I j (Q) will match. w This property can be generalized to other transformation groups. w Each line segment is a basis of a coordinate system Translation, Rotation, and Orientation defined I(P) = transform all other segments into new CS Cohen and Guibas

20. Notes on GH w Each segment will be transformed to 2m Coordinate systems w I(P) stores O(m 2 ) segments w Can be done as preprocessing step w Expressing the different possible transformations using each segment as a basis Cohen and Guibas

21. Querying the Database w Query image undergoes the feature extraction process w For each query feature, a nearest-neighbor query is applied and the k closest or within some j w Similarity score increases if database image has a feature that is “close” to the query feature Cohen and Guibas

22. “Closeness” w How do you describe the closeness of two lines? w Transform to a 4D space made of (l, ,a,b) w With two (l, ,a,b) descriptions for lines, can compute distance w Divide by standard deviation over sample of database features

23. Details w Closeness is relative to database contents w Nearest-neighbor algorithm by Arya, Mount, et. al (1994). Query time for k nearest features is O(k log n) Cohen and Guibas

24. Advantages w Fast Queries database of 500 characters in 1 second on SGI Indy w Queries based on important features Disadvantages w Working domain currently limited w Could get too expensive as complexity in images increases Cohen and Guibas

25. A Multi-Resolution Technique for Comparing Images Using the Hausdorff Distance Daniel Huttenlocher and William Rucklidge 1992Cornell University Huttenlocher and Rucklidge

26. Directed Hausdorff Distance w Given A={a 1, …, a m } and B={b 1, …, b m } w Identifies the point in A farthest from any point in B w Measures the degree of mismatch between between two sets.

27. Properties of Hausdorff Distances w Not symmetric h(A,B) != h(B,A) w Compute k th maximum Notion of rank Reduces sensitivity Fraction of A within h(A,B) of B Obscured portions w h(A,B) = hypothesis h(B,A) = test

28. Transformations t( ) = w Given A is an image, B is the model w Without Orientation, if A is in B then A undergoes transformation t. f B (t)=H(t(B),A) t=(t x,t y,s x,s y )forward f A (t)=H(A,t(B)) reverse

29. Bidirectional Hausdorff Distance w Solve for which values of t, the following holds: w Results in searching a four dimensional space

30. Restricting Search Space w Slope of f(t)=H LK (A,t(B)) is linear w Divide space into cells w Calculate H LK (A,t c (B)) w Determine a maximum delta per cell Based on limit in scale and translation Allows for quick rejection and acceptance w Label cells as interesting or disregard

31. Restricting Search Space w Create smaller cells from interesting cells w Bounds based on transformations w Quickly narrow down to areas that could possibly be within  of A

32. Subtleties w Discretization useful if working in computer vision domain (integers) w Can compare partially obscured images w Optimizations Early rejection/acceptance w Pretty slow 200 to 250 seconds Website on submarines

33. Huttenlocher and Rucklidge

34. Advantages w Accurate w Geared towards image processing and vision w Partially obscured images w Searches similar to humans Disadvantages w Slow w No Orientation w Database must be specialized w Potential problems in generating queries

35. Recap w Three Algorithms Color Based Color Signatures Earth Mover’s Distance Shaped Based Polylines Transform Invariant Sets Vision Based Hausdorff Distance Subdivision of Transformation Space www.sportsmanscaps.com

36. Final Thoughts w Algorithms work well in various domains w Query construction not formalized w Other methods: wavelet-based texture-based object-based w Took 5 minutes to find “Shrine to Men with Long Hair” and “Videos of Women Getting Their Head Shaved” www.jerryspringer.com All other images generated by author using Paint Shop Pro