Survey of Algorithms to Query Image Databases COMP :Computational Geometry Benjamin Lok 11/2/98 Image from Kodak’s PhotoQuilt

Outline of Talk w Overview of the problem w Three methods Color based Shape based Vision based w Conclusions Image from Microsoft Clip Gallery

Problem w Query an image database w What does a “match” mean? Application dependent Notion of subjectivity Sensitivity to noise

Problem w Semantic similarity is still not possible ex. “All images with cats” w To determine similarity, we need to a new: Metric Space Images from Microsoft Clip Gallery and website

Ambiguity Guy Girl Guy Images from “Shrine to Long Haired Men” and “Videos of Women Getting Their Heads Shaved” websites

Yossi Rubner, Leonidas Guibas, Carlos Tomasi (1997) Stanford Vision Laboratory The Earth Mover’s Distance, Multi-Dimensional Scaling, and Color-Based Image Retrieval Image from Microsoft Clip Gallery

Color Signatures w Utilize the CIE-LAB color space Based on human perception of color w Map each pixel to a point in color space Common color values increase weight of point w Group clusters into points (8-12 per image) Rubner, Guibas, and Tomasi

Earth Mover’s Distance w To compare two images, compute the “work” needed to move the cluster points from one image to the other Rubner, Guibas, and Tomasi

Earth Mover’s Distance (cont) w Solving a linear programming problem: Given two signatures: p = {(p 1,w p1 ),…,(p m,w pm )} and q = {(q 1,w q1 ),…,(q n,w qn )} Find C where C ij is the amount of weight p i matched q j

Applications w Visualize Databases (Queries and Results) w Scale the multiple dimensions into 2D using MDS and minimize STRESS Rubner, Guibas, and Tomasi

Database Visualization Rubner, Guibas, and Tomasi

Algorithm Recap w Map pixels to 3D color space points w Locate and compress “clusters” of points w 8 to 12 points determine the color signature w Calculate the Earth Mover’s Distance to determine “distance” between two images Image from YenPen Stationary Website

Advantages w Based on human perception of color w Some invariance to small change in viewpoint and lighting w Meaningful metric w Relatively fast w Can embed multiple metrics Disadvantages w Depending on application, query format might be not be intuitive w Not much use for non- color images w False positives a real possibility depending on working domain

Shaped-based Image Retrieval Using Geometric Hashing Scott D. Cohen and Leonidas J. Guibas 1997 Stanford Vision Laboratory Image from Microsoft Clip Gallery

Overview w Implementation Search through 500 Chinese characters w Goals Provide invariance to scale, rotation, and translation speed and accuracy Cohen and Guibas

Generating a Illustration w Illustration - set of curves that summarize an image Edgel detection Medial Axis determination Cohen and Guibas

Approximating with Polylines w Convert medial axis representation to polylines w Tradeoff between speed and accuracy Cohen and Guibas

Geometric Hashing w Geometric Hashing - method used to compare two point sets under some transformation group w Take each point and use it as the origin of a coordinate system Cohen and Guibas

Geometric Hashing (cont) w If translating P by q j - p i produces a good match I i (P) and I j (Q) will match. w This property can be generalized to other transformation groups. w Each line segment is a basis of a coordinate system Translation, Rotation, and Orientation defined I(P) = transform all other segments into new CS Cohen and Guibas

Notes on GH w Each segment will be transformed to 2m Coordinate systems w I(P) stores O(m 2 ) segments w Can be done as preprocessing step w Expressing the different possible transformations using each segment as a basis Cohen and Guibas

Querying the Database w Query image undergoes the feature extraction process w For each query feature, a nearest-neighbor query is applied and the k closest or within some j w Similarity score increases if database image has a feature that is “close” to the query feature Cohen and Guibas

“Closeness” w How do you describe the closeness of two lines? w Transform to a 4D space made of (l, ,a,b) w With two (l, ,a,b) descriptions for lines, can compute distance w Divide by standard deviation over sample of database features

Details w Closeness is relative to database contents w Nearest-neighbor algorithm by Arya, Mount, et. al (1994). Query time for k nearest features is O(k log n) Cohen and Guibas

Advantages w Fast Queries database of 500 characters in 1 second on SGI Indy w Queries based on important features Disadvantages w Working domain currently limited w Could get too expensive as complexity in images increases Cohen and Guibas

A Multi-Resolution Technique for Comparing Images Using the Hausdorff Distance Daniel Huttenlocher and William Rucklidge 1992Cornell University Huttenlocher and Rucklidge

Directed Hausdorff Distance w Given A={a 1, …, a m } and B={b 1, …, b m } w Identifies the point in A farthest from any point in B w Measures the degree of mismatch between between two sets.

Properties of Hausdorff Distances w Not symmetric h(A,B) != h(B,A) w Compute k th maximum Notion of rank Reduces sensitivity Fraction of A within h(A,B) of B Obscured portions w h(A,B) = hypothesis h(B,A) = test

Transformations t( ) = w Given A is an image, B is the model w Without Orientation, if A is in B then A undergoes transformation t. f B (t)=H(t(B),A) t=(t x,t y,s x,s y )forward f A (t)=H(A,t(B)) reverse

Bidirectional Hausdorff Distance w Solve for which values of t, the following holds: w Results in searching a four dimensional space

Restricting Search Space w Slope of f(t)=H LK (A,t(B)) is linear w Divide space into cells w Calculate H LK (A,t c (B)) w Determine a maximum delta per cell Based on limit in scale and translation Allows for quick rejection and acceptance w Label cells as interesting or disregard

Restricting Search Space w Create smaller cells from interesting cells w Bounds based on transformations w Quickly narrow down to areas that could possibly be within  of A

Subtleties w Discretization useful if working in computer vision domain (integers) w Can compare partially obscured images w Optimizations Early rejection/acceptance w Pretty slow 200 to 250 seconds Website on submarines

Huttenlocher and Rucklidge

Advantages w Accurate w Geared towards image processing and vision w Partially obscured images w Searches similar to humans Disadvantages w Slow w No Orientation w Database must be specialized w Potential problems in generating queries

Recap w Three Algorithms Color Based Color Signatures Earth Mover’s Distance Shaped Based Polylines Transform Invariant Sets Vision Based Hausdorff Distance Subdivision of Transformation Space

Final Thoughts w Algorithms work well in various domains w Query construction not formalized w Other methods: wavelet-based texture-based object-based w Took 5 minutes to find “Shrine to Men with Long Hair” and “Videos of Women Getting Their Head Shaved” All other images generated by author using Paint Shop Pro