IIIT Hyderabad Learning in Large Scale Image Retrieval Systems Under the guidance of: Dr. C. V. Jawahar & Dr. Vikram Pudi by Pradhee Tandon Roll No

IIIT Hyderabad Image Retrieval Explosive growth in images Easy access to most of these on the web Contemporary systems used tags The best commercial systems are still tag based Inadequate and unreliable Manual tagging is infeasible Content based retrieval is the best option

IIIT Hyderabad Content Based Image Retrieval Image Feature Database Feature Extraction Comparison Module Query Results QueryResults

IIIT Hyderabad Content Based Image Retrieval Image Feature Database Feature Index Feature Extraction Comparison Module Query Results

IIIT Hyderabad Content Based Image Retrieval Feature Extraction Comparison Module Query Results Feature Index Relevance Learning Memory & Logs Rf

IIIT Hyderabad Content Based Image Retrieval Feature Extraction Comparison Module Query Results Feature Index Relevance Learning Learning Memory Rf

IIIT Hyderabad Scope of work Features Color Histograms Texture Filters Shape Context SIFT GLOH Spatial indexing methods Kd – trees R-tree Distance Metrics Euclidean Mahalanobis KL Divergence Relevance Feedback Short term learning Long term learning Content Free Retrieval Active Learning Diversity Retrieval

IIIT Hyderabad Requirements Efficient Image Retrieval Learning relevant features in images Learning image-image relationships Diversity in retrieval for improved learning

IIIT Hyderabad Requirements Efficient Image Retrieval Learning relevant features in images Learning image-image relationships Diversity in retrieval for improved learning

IIIT Hyderabad FISH – The System

IIIT Hyderabad Implementation of FISH Image Representation in FISH MPEG-7 Colour Structure Descriptor Maximum Response Filters for Textures Developed on the LAMP stack, using C/C++, Perl, PHP, HTML, MySQL and Apache TPIE toolkit from Duke University for B+ tree implementation

IIIT Hyderabad Indexing Scheme Interactive response over large databases (less than a second) Efficient scalable index (dynamic with data) Similarity indexing scheme (r-tree, kd-tree, ss-tree) Support for changing similarity metrics (metric changes with learning) B+ tree based index Nataraj et. al, MMM 2007, Efficient Search with Changing Similarity Measures on Large Multimedia Datasets

IIIT Hyderabad The Retrieval Algorithm Feature Extraction Comparison Module Query Results Feature Index Relevance Learning Learning Memory Rf Retrieval in FISH

IIIT Hyderabad Retrieval Performance Retrieval times with increasing #Dimensions in ( secs ) & DB size fixed at 1 lakh Retrieval times with increasing DB size in ( secs ) & #dimensions fixed at 10

IIIT Hyderabad Requirements Efficient Image Retrieval Learning relevant features in images Learning image-image relationships Diversity in retrieval for improved learning

IIIT Hyderabad Content Based Image Retrieval Feature Extraction Comparison Module Query Results Feature Index Relevance Learning Memory & Logs Rf

IIIT Hyderabad Learning - expectations Effective – capture user intent correctly Efficient – interactive retrieval response Scalable – limited computational overhead for large collections Adaptive – caters to individual user’s subjectivities Intra-query or short term learning (STL) Evolving – incrementally improves across users and queries Inter-query or long term learning (LTL) Dynamic – seamlessly absorbs changes in the collection

IIIT Hyderabad Learning - Method Relative relevance of features using feedback Numerous methods can be used Discriminative variance is as - Weights are incrementally learnt over iterations using – At the end of the session long term learning is updated for the relevant images using – Image to image dissimilarity is computed using – Weighted Mahalanobis

IIIT Hyderabad Improved accuracy Precision across sessions using LTL Rank Convergence of top N relevant samples Sum of ranks of Top 10 relevant images converges close zero (downshifted) over multiple sessions with long term learning

IIIT Hyderabad Improved retrieval System learns the yellow flower in the hedge over sessions System learns the rock and sky pattern over sessions Top 9 results for queries across 3 different sessions (left-most are queries too)

IIIT Hyderabad Optimized Retrieval

IIIT Hyderabad Long term memory allows learning of relevant image features Converges to popular content over sessions For example, Assume, features are associated with individual pixels, colors Consider a gray image, pixels for more relevant features are colored brighter Content from Learning Actual imageContent image

IIIT Hyderabad Visual Content Extraction After a large number of sessions Over sessions

IIIT Hyderabad Requirements Efficient Image Retrieval Learning relevant features in images Learning image-image relationships Diversity in retrieval for improved learning

IIIT Hyderabad Content Based Image Retrieval Feature Extraction Comparison Module Query Results Feature Index Relevance Learning Memory & Logs Rf

IIIT Hyderabad Image-Image Relations Query Given a history of patterns in behavior and a current partial pattern, collaborative filtering predicts the next pattern for the latter Content Free Image Retrieval or CFIR, uses feedback logs to predict the next set of results for the current pattern

IIIT Hyderabad Hybrid Image Retrieval We integrate them in a Bayesian inference like framework, –The a priori relationships from logs –The evidence from visual similarity –Retrieval is an a posteriori estimation problem CBIRCFIR Semantic gapNo semantic gap No cold startCold start Handles unseen queriesCannot handle unseen queries Not affected by sparsenessAffected by sparseness

IIIT Hyderabad Bayesian Image Retrieval System Architecture of the proposed Bayesian Image Retrieval System

IIIT Hyderabad Bayesian Image Retrieval... posterior = prior * evidence Efficient a priori updates –The prior probabilities are not stored, reduces updates –Co-relevance between images are stored in a matrix –The a priori is estimated using the co-relevance values Evidence computation –Weights are learnt using discriminative variance method –Weighted Mahalanobis for (dis)similarity

IIIT Hyderabad Concept Discovery a priori matrix has embedded patterns of similar co-relevances Co-relevance patterns can be summarized into ‘k’ concepts –cluster the patterns into V concepts 1…k. –clustering is repetitive but offline –exhaustive comparisons are avoided

IIIT Hyderabad Accuracy with Bayesian Gain in precision with Bayesian Using real human user feedback logs Using annotation based feedback logs Gain in precision across sessions

IIIT Hyderabad Accuracy with Bayesian CBIR results and Bayesian results

IIIT Hyderabad Requirements Efficient Image Retrieval Learning relevant features in images Learning image-image relationships Diversity in retrieval for improved learning

IIIT Hyderabad Diversity in Image Retrieval Query

IIIT Hyderabad Skylines – the natural solution Results should be similar in a variety of different ways Skylines return non-dominated samples Non-dominated samples are closer to the query than all the others, in at-least one way (attribute)

IIIT Hyderabad Skyline Extraction Architecture of the proposed skyline based similarity retrieval system

IIIT Hyderabad Diversity with Skylines

IIIT Hyderabad Efficient Skylines Real image data with 12 and 9 dimensions with real images Synthetic data with 10 dimensions and and data points

IIIT Hyderabad Preferential Skylines Relevance feedback represents user’s preference Weights learned using feature relevance Skylines are then computed in user space

IIIT Hyderabad Designed and implemented a web- based image retrieval system, called FISH Proposed an efficient feature relevance learning algorithm Integration of complimentary CFIR and CBIR a Bayesian inference framework Skylines to retrieve diversely similar samples for a given query Contributions

IIIT Hyderabad Future directions Videos are richer and the next step Efficient higher level concept discovery is needed Skylines with preference should be explored further

IIIT Hyderabad Publications Pradhee Tandon, Piyush Nigam, Vikram Pudi, C. V. Jawahar, “FISH: A Practical System for Fast Interactive Image Search in Huge Databases”, in Proceedings of the 7th ACM International Conference on Image and Video Retrieval (CIVR ’08), July 6-8, 2008, Niagara Falls, Canada. Pradhee Tandon, C. V. Jawahar, “Long Term Learning for Content Extraction in Image Retrieval”, in Proceedings of the 15th National Conference on Communications (NCC ’09), January 16-18, 2009, Guwahati, India. Pradhee Tandon, C. V. Jawahar, “Bayesian Image Retrieval” submitted to 3rd International Conference on Pattern Recognition and Machine Intelligence (PReMI ’09), December 16-20, 2009, New Delhi, India.

IIIT Hyderabad Thank you

IIIT Hyderabad Addendum

IIIT Hyderabad The Retrieval Algorithm *Learning discussed in detail later

IIIT Hyderabad Bayesian Image Retrieval The a priori probability of retrieving image ‘a’ with query ‘q’ is P(R) = n(q,a)/n(a) –where n(a) denotes relevant retrievals for ‘a’ The evidence from visual similarity is computed as p(S|R) = f(w,q,a) –where weights ‘w’ are refined using relevance feedback The posterior probability of retrieval is computed as p(R|S) = p(S|R) P(R) –the denominator can be ignored –PicHunter is a hybrid but does no feature learning –Zhong et. al, use Bayes inference for a probabilistic decision only

IIIT Hyderabad Skyline Extraction