PageRank for Product Image Search Yushi Jing, Shumeet Baluja College of Computing, Georgia Institute of Technology Google, Inc. WWW 2008 Referred Track: Rich Media Summarized and Presented by Seungseok Kang, IDS Lab.

Copyright  2008 by CEBT Outline  Introduction  Background and Related Work  Approach and Algorithm Features Generation and Representation Query Dependent Ranking  Full Retrieval System Queries with Homogeneous Visual Concepts Queries with Heterogeneous Visual Concepts  Experimental Results  Conclusion

Copyright  2008 by CEBT Introduction  Image search has become a popular feature in search engines Yahoo, MSN, Google  The majority of common image search Based on the text on the pages in which the image is embedded – Text in the body of the page, Anchor-text, Image name, etc. Text-based search of web pages is a well studied problem Fundamental task of image analysis is yet unsolved Image processing required can be quite expensive  PageRank for the image search Analyzing the distribution of visual similarities among the images Finding the multiple visual themes and their relative strengths in a large set of images

Copyright  2008 by CEBT Eiffel Tower vs. McDonalds

Copyright  2008 by CEBT Challenge Issues  The concept of inferring common visual themes to creating a scalable and effective algorithm Image processing – The goal of query is to find what is common among the images – The common features may occur anywhere in the images – Local features Utilization of information – Simple counting will yield poor results – Inferring a graph between the images where images are linked to each other based on their similarity

Copyright  2008 by CEBT Background and Related Work  Object Category Model Trained from the top search results Re-rank images based on their fit to the model Lack of heterogeneous image search – E.g. “Apple” “Nemo” “Jaguar”  Intuitive graph-model Based on the content-based image ranking Using expected user behavior – Visual similarities Treating images as web documents Estimating the likelihood of images visited by a user traversing through the visual- hyperlinks  Similarity based graph For semi-supervised learning

Copyright  2008 by CEBT Contribution  Introducing a novel, simple algorithm to rank images based on their visual similarities  Introducing a system to re-rank current Google image search results Similarity score among images can be derived from a comparison of their local descriptors  Improving the image search result for queries that are of the most interest to a large set of people significantly

Copyright  2008 by CEBT Approach and Algorithm  Preliminaries Eigenvector Centrality – A sort of a square stochastic adjacency matrix – Providing a principled method to combine the “importance” of a vertex with those of its neighbors in ranking – PageRank pre-computes a rank vector to estimate the importance for all of the Web pages Random Walk explanation – The ranking scores correspond to the likelihood of arriving in each of the vertices by traversing through the graph with a random starting – If a user is viewing an image, other related (similar) images may also be of interest (jumping to other random image)

Copyright  2008 by CEBT Approach and Algorithm (cont’d)  Image rank (IR) S* is the column normalized, symmetrical adjacency matrix S where S u,v measures the visual similarities Iterative IR yields the dominant eigenvector of the matrix S*  Image rank with random walk d is damping factor (commonly d > 0.8 in practice) Considering a small probability for a random walk to go to some other images which is not connected in the graph  Then, how can we calculate visual similarities?

Copyright  2008 by CEBT Features generation and representation  A reliable measure of image similarity Global features are often too restrictive (“Prius”) – Color histograms – Shape analysis Local descriptors are useful – Contains a richer set of image information – Relatively stable under different transformation – A type of local descriptors Harris corners Scale Invariant Feature Transform (SIFT) Shape Context Spin Images

Copyright  2008 by CEBT Local Descriptors  SIFT with a Different of Gaussian (DoG) Extracting a features of images without regarding to scale and rotation Simple process – 1. Converting original image into grayscale – 2. Applying Gaussian Filter – 3. Finding DoG interest point (candidate keypoint) – 4. pruning the candidate keypoint – 5. deriving description vectors – 6. comparing the description vectors  The similarity is defined as the number interest point (keypoint) shared between two images divided by their average number of interest points

Copyright  2008 by CEBT Query Dependent Ranking  Generating the similarity graph S is computationally infeasible for the billions of images Need to reduce the computational cost  Query dependent ranking A practical method to obtain the initial set of candidates Rely on the existing commercial search engine for the initial grouping of semantically similar images – Ex) given the query “Eiffel Towers” – Extracting the top-N results from existing search engines – Creating the graph of visual similarity on the N images – Computing the image rank only on this subset  Then, how can this approach improve the relevancy and diversity of image search results? Query Dependent

Copyright  2008 by CEBT A Full Retrieval System  The goal of image-search engines Retrieving image results that are relevant to the query and diverse enough to cover variations of visual of semantic concepts – “Without the analyzing the content of images, there is no reliable way to actively promote the diversity of the results”  Queries with homogeneous visual concepts “Mona-Lisa”, “Eiffel Tower”, “Albert Einstein” Achieved by identifying the vertices that are located at the “center” of weighted similarity graph  Queries with heterogeneous visual concepts “Jaguar”, “Apple”, “Monet Painting” The approach is able to identify a relevant and diverse set of images as top ranking results Simple heuristics can help for analyzing the graph

Copyright  2008 by CEBT Homogeneous Concept: Example

Copyright  2008 by CEBT Heterogeneous Concept: Example

Copyright  2008 by CEBT Heterogeneous Concept: Example

Copyright  2008 by CEBT Experimental Results  Test set 2000 most popular product queries on Google – Google product search: “ipod”, “xbox”, “Picasso”, “Fabreze”, …… – Extracted the top 1000 search results from Google Image Search – Fewer than 5% of the images had at least 1 connection – Concentrated on the approximately 1000 remaining queries  Challenge issues Quantifying the quality of sets of image is very hard – User preference to an image is heavily influenced by a user’s personal tastes and biases – Asking the user to compare the quality of a set of images is difficult and time consuming task – Assessing the differences in ranking is error-prone and imprecise  Two evaluation strategies Minimizing irrelevant images Click studies

Copyright  2008 by CEBT Experimental Results (cont’d)  Minimizing irrelevant images For studying a conservative version of “relevancy” of the ranking results – Asking the user: “Which of the images are the least relevant?” IR>Google: 762 Google>IR: 70 Google=IR: 202  Click Study User satisfaction is not purely a function of relevance – Users usually click in the images they are interested in An effective way to measure search quality is to analyze the total number of “clicks” each image receives – Collect clocks for the top 40 images on 130 common queries – The image selected by IR to be in the top-20 would have received approximately 17.5% more clicks than those in the default ranking In case of Inflated logos Screenshots of Web pages In case of Inflated logos Screenshots of Web pages

Copyright  2008 by CEBT Conclusion  Proposed a simple mechanism to incorporate the advancement made in using link and network analysis for Web-document search into image search Image Ranking  Demonstrated an effective method to infer a graph in which the images could be embedded Visual similarities with visual-hyperlinks Rely on human knowledge and the intelligence of crowds  Proposed the ability to customize the similarity function based on the expected distribution of queries  Future work Determining the performance of the system under adversarial condition Studying about the role of duplicate and near-duplicate images in terms of the potential for biasing the approach and transitional probabilities