1. IIIT Hyderabad Learning in Large Scale Image Retrieval Systems Under the guidance of: Dr. C. V. Jawahar & Dr. Vikram Pudi by Pradhee Tandon Roll No. 200607020

2. IIIT Hyderabad Image Retrieval Explosive growth in images Easy access to most of these on the web Contemporary systems used tags The best commercial systems are still tag based Inadequate and unreliable Manual tagging is infeasible Content based retrieval is the best option

3. IIIT Hyderabad Content Based Image Retrieval Image Feature Database Feature Extraction Comparison Module Query Results QueryResults

4. IIIT Hyderabad Content Based Image Retrieval Image Feature Database Feature Index Feature Extraction Comparison Module Query Results

5. IIIT Hyderabad Content Based Image Retrieval Feature Extraction Comparison Module Query Results Feature Index Relevance Learning Memory & Logs Rf

6. IIIT Hyderabad Content Based Image Retrieval Feature Extraction Comparison Module Query Results Feature Index Relevance Learning Learning Memory Rf

7. IIIT Hyderabad Scope of work Features Color Histograms Texture Filters Shape Context SIFT GLOH Spatial indexing methods Kd – trees R-tree Distance Metrics Euclidean Mahalanobis KL Divergence Relevance Feedback Short term learning Long term learning Content Free Retrieval Active Learning Diversity Retrieval

8. IIIT Hyderabad Requirements Efficient Image Retrieval Learning relevant features in images Learning image-image relationships Diversity in retrieval for improved learning

9. IIIT Hyderabad Requirements Efficient Image Retrieval Learning relevant features in images Learning image-image relationships Diversity in retrieval for improved learning

10. IIIT Hyderabad FISH – The System

11. IIIT Hyderabad Implementation of FISH Image Representation in FISH MPEG-7 Colour Structure Descriptor Maximum Response Filters for Textures Developed on the LAMP stack, using C/C++, Perl, PHP, HTML, MySQL and Apache TPIE toolkit from Duke University for B+ tree implementation

12. IIIT Hyderabad Indexing Scheme Interactive response over large databases (less than a second) Efficient scalable index (dynamic with data) Similarity indexing scheme (r-tree, kd-tree, ss-tree) Support for changing similarity metrics (metric changes with learning) B+ tree based index Nataraj et. al, MMM 2007, Efficient Search with Changing Similarity Measures on Large Multimedia Datasets

13. IIIT Hyderabad The Retrieval Algorithm Feature Extraction Comparison Module Query Results Feature Index Relevance Learning Learning Memory Rf Retrieval in FISH

14. IIIT Hyderabad Retrieval Performance Retrieval times with increasing #Dimensions in ( secs ) & DB size fixed at 1 lakh Retrieval times with increasing DB size in ( secs ) & #dimensions fixed at 10

15. IIIT Hyderabad Requirements Efficient Image Retrieval Learning relevant features in images Learning image-image relationships Diversity in retrieval for improved learning

16. IIIT Hyderabad Content Based Image Retrieval Feature Extraction Comparison Module Query Results Feature Index Relevance Learning Memory & Logs Rf

17. IIIT Hyderabad Learning - expectations Effective – capture user intent correctly Efficient – interactive retrieval response Scalable – limited computational overhead for large collections Adaptive – caters to individual user’s subjectivities Intra-query or short term learning (STL) Evolving – incrementally improves across users and queries Inter-query or long term learning (LTL) Dynamic – seamlessly absorbs changes in the collection

18. IIIT Hyderabad Learning - Method Relative relevance of features using feedback Numerous methods can be used Discriminative variance is as - Weights are incrementally learnt over iterations using – At the end of the session long term learning is updated for the relevant images using – Image to image dissimilarity is computed using – Weighted Mahalanobis

19. IIIT Hyderabad Improved accuracy Precision across sessions using LTL Rank Convergence of top N relevant samples Sum of ranks of Top 10 relevant images converges close zero (downshifted) over multiple sessions with long term learning

20. IIIT Hyderabad Improved retrieval System learns the yellow flower in the hedge over sessions System learns the rock and sky pattern over sessions Top 9 results for queries across 3 different sessions (left-most are queries too)

21. IIIT Hyderabad Optimized Retrieval

22. IIIT Hyderabad Long term memory allows learning of relevant image features Converges to popular content over sessions For example, Assume, features are associated with individual pixels, colors Consider a gray image, pixels for more relevant features are colored brighter Content from Learning Actual imageContent image

23. IIIT Hyderabad Visual Content Extraction After a large number of sessions Over sessions

24. IIIT Hyderabad Requirements Efficient Image Retrieval Learning relevant features in images Learning image-image relationships Diversity in retrieval for improved learning

25. IIIT Hyderabad Content Based Image Retrieval Feature Extraction Comparison Module Query Results Feature Index Relevance Learning Memory & Logs Rf

26. IIIT Hyderabad Image-Image Relations Query Given a history of patterns in behavior and a current partial pattern, collaborative filtering predicts the next pattern for the latter Content Free Image Retrieval or CFIR, uses feedback logs to predict the next set of results for the current pattern

27. IIIT Hyderabad Hybrid Image Retrieval We integrate them in a Bayesian inference like framework, –The a priori relationships from logs –The evidence from visual similarity –Retrieval is an a posteriori estimation problem CBIRCFIR Semantic gapNo semantic gap No cold startCold start Handles unseen queriesCannot handle unseen queries Not affected by sparsenessAffected by sparseness

28. IIIT Hyderabad Bayesian Image Retrieval System Architecture of the proposed Bayesian Image Retrieval System

29. IIIT Hyderabad Bayesian Image Retrieval... posterior = prior * evidence Efficient a priori updates –The prior probabilities are not stored, reduces updates –Co-relevance between images are stored in a matrix –The a priori is estimated using the co-relevance values Evidence computation –Weights are learnt using discriminative variance method –Weighted Mahalanobis for (dis)similarity

30. IIIT Hyderabad Concept Discovery a priori matrix has embedded patterns of similar co-relevances Co-relevance patterns can be summarized into ‘k’ concepts –cluster the patterns into V concepts 1…k. –clustering is repetitive but offline –exhaustive comparisons are avoided

31. IIIT Hyderabad Accuracy with Bayesian Gain in precision with Bayesian Using real human user feedback logs Using annotation based feedback logs Gain in precision across sessions

32. IIIT Hyderabad Accuracy with Bayesian CBIR results and Bayesian results

33. IIIT Hyderabad Requirements Efficient Image Retrieval Learning relevant features in images Learning image-image relationships Diversity in retrieval for improved learning

34. IIIT Hyderabad Diversity in Image Retrieval Query

35. IIIT Hyderabad Skylines – the natural solution Results should be similar in a variety of different ways Skylines return non-dominated samples Non-dominated samples are closer to the query than all the others, in at-least one way (attribute)

36. IIIT Hyderabad Skyline Extraction Architecture of the proposed skyline based similarity retrieval system

37. IIIT Hyderabad Diversity with Skylines

38. IIIT Hyderabad Efficient Skylines Real image data with 12 and 9 dimensions with 11901 real images Synthetic data with 10 dimensions and 10000 and 15000 data points

39. IIIT Hyderabad Preferential Skylines Relevance feedback represents user’s preference Weights learned using feature relevance Skylines are then computed in user space

40. IIIT Hyderabad Designed and implemented a web- based image retrieval system, called FISH Proposed an efficient feature relevance learning algorithm Integration of complimentary CFIR and CBIR a Bayesian inference framework Skylines to retrieve diversely similar samples for a given query Contributions

41. IIIT Hyderabad Future directions Videos are richer and the next step Efficient higher level concept discovery is needed Skylines with preference should be explored further

42. IIIT Hyderabad Publications Pradhee Tandon, Piyush Nigam, Vikram Pudi, C. V. Jawahar, “FISH: A Practical System for Fast Interactive Image Search in Huge Databases”, in Proceedings of the 7th ACM International Conference on Image and Video Retrieval (CIVR ’08), July 6-8, 2008, Niagara Falls, Canada. Pradhee Tandon, C. V. Jawahar, “Long Term Learning for Content Extraction in Image Retrieval”, in Proceedings of the 15th National Conference on Communications (NCC ’09), January 16-18, 2009, Guwahati, India. Pradhee Tandon, C. V. Jawahar, “Bayesian Image Retrieval” submitted to 3rd International Conference on Pattern Recognition and Machine Intelligence (PReMI ’09), December 16-20, 2009, New Delhi, India.

43. IIIT Hyderabad Thank you

44. IIIT Hyderabad Addendum

45. IIIT Hyderabad The Retrieval Algorithm *Learning discussed in detail later

46. IIIT Hyderabad Bayesian Image Retrieval The a priori probability of retrieving image ‘a’ with query ‘q’ is P(R) = n(q,a)/n(a) –where n(a) denotes relevant retrievals for ‘a’ The evidence from visual similarity is computed as p(S|R) = f(w,q,a) –where weights ‘w’ are refined using relevance feedback The posterior probability of retrieval is computed as p(R|S) = p(S|R) P(R) –the denominator can be ignored –PicHunter is a hybrid but does no feature learning –Zhong et. al, use Bayes inference for a probabilistic decision only

47. IIIT Hyderabad Skyline Extraction