Discriminative Relevance Feedback With Virtual Textual Representation For Efficient Image Retrieval Suman Karthik and C.V.Jawahar

Multimedia data is growing exponentially.  Cheap high quality digital imaging devices  Sharing of multimedia data on the internet Content based organization and retrieval is a viable way of accessing this data. Introduction

What do users look for? *In CBIR systems users look for ‘things’ not ‘stuff’.  More than global image properties  Traditional object recognition won’t work  Two choices: Rely on text to identify objects Look at regions (objects or parts of objects) The region based image retrieval paradigm was successfully applied by Carson et al. in Blobworld[ ] and Wang et al. in SIMPLIcity[2001] *Chad Carson: Blobworld

Practical CBIR systems  Practical large scale deployment of CBIR systems require.  Efficient indexing and retrieval of thousands of documents.  Flexible framework for retrieval based on various types of features. For example Specialized features for highly specific sub domains like faces, vehicles, monuments.  Ability to scale up to millions of images without a significant performance trade off.

Lessons From Text Retrieval Large scale text retrieval systems have been successfully deployed.  Search Engines like Efficient indexing and retrieval of millions of documents has been achieved. The text retrieval frameworks are adaptive enough to be applied to specialized domains.

Images as text documents. Color (YUV), compactness and location of segment are used to encode the segment as text. Transformation Virtual Textual Representation Document Image Words SegmentsText Segmentation

Transformation Feature Space Bins represented by strings or words Quantization Color Compactness Position

Transformation Quantization can be achieved in a number of ways.  Uniform vector space quantization for data set with a uniform feature point distribution.  Density based quantization of the feature space can be achieved with simple k-means quantization. Irrespective of the quantization applied each cell in the vector space has a representative string. Each image segment is assigned to a cell and is assigned the representative string of the cell.

Discriminative Relevance Feedback Discriminative regions are given higher weight than representative regions.  Image segments that can differentiate between roses and other flowers are given higher weight with respect to the class roses. Regions aiding classification rather than clustering are chosen.  Image segments containing humans are able to differentiate between ‘surfer’ and ‘wave’ images.

Transformation Intuitive way of learning content Over segmentation and subsequent deduction of content can be achieved if the problem is modeled like this. Document Image Words SegmentsText Segmentation

Perfromance Discriminative relevance feedback consistently out performed Region based importance method. Given are the precision data for discriminative relevance feedback and Bayesian region importance relevance feedback.

Indexing CBIR systems usually use spatial databases to index and retrieve data.  Blobworld uses variants or R-trees. *[ Megan Thomas, Chad Carson, Joseph M. Hellerstein Creating a Customized Access Method for Blobworld (2000) ICDE ] Relevance feedback skews the feature space rendering spatial databases inefficient. *[ peng et al. Kernel Indexing for Relevance Feedback Image Retrieval ]

Elastic Bucket Trie Null A B C A B A B B Nodes Buckets CAB CBA Overflow Split A B Query BBC Retrieved Bucket Insert

Spatial Data Structures  Become inefficient when used with relevance feedback.  Requires costly arithmetic operations.  Number of splits of the spatial data structure is not fixed  Strictly adheres to spatial characteristics of the feature space. Spatial data structures VS EBT Elastic Bucket Trie  Not effected by relevance feedback schemes.  Requires bit opertations  Number of splits of EBT is limited.  The trie need not adhere to an underlying spatial structure though that is also possible.* Suman Karthik, C.V. Jawahar, Efficient Region Based Indexing and Retrieval for Images with Elastic Bucket Tries, ICPR(2006)

Relevance feedback and EBT Typical relevance feedback algorithms need to be modified to work with text. Keywords emerge with relevance feedback signifying association between key segments. EBT can be used without any modifications with discriminative relevance feedback.

Bag of words The scheme is very similar to contemporary bag of words approaches. Interest point based bag of words approaches can also be adapted to work within our framework.* Any type of vector quantization of the feature space used by these schemes can use EBT. *R. Fergus, L. Fei-Fei, P. Perona, and A. Zisserman. Learning object categories from google's image search. ICCV, 2005.

Future Work Usage of heterogeneous strings to describe an image. Text encoding of images that is highly distinctive. Text encoding of images that is robust to segmentation inaccuracies. Text based image mining to discover concepts and their features.

THE END