1. Relevance Feedback based on Parameter Estimation of Target Distribution K. C. Sia and Irwin King Department of Computer Science & Engineering The Chinese University of Hong Kong 15 May IJCNN 2002

2. Relevance Feedback Based on Parameter Estimation of Target Distribution Agenda Introduction to content based image retrieval (CBIR) and relevance feedback (RF) Introduction to content based image retrieval (CBIR) and relevance feedback (RF) Former approaches Former approaches Tackling the problem Tackling the problem Parameter estimation of target distribution Parameter estimation of target distribution Experiments Experiments Future works and conclusion Future works and conclusion

3. Relevance Feedback Based on Parameter Estimation of Target Distribution Content Based Image Retrieval How to represent an image? How to represent an image? Feature extraction Feature extraction Colour histogram (RGB) Colour histogram (RGB) Co-occurrence matrix texture analysis Co-occurrence matrix texture analysis Shape representation Shape representation Feature vector Feature vector Map images to points in hyper-space Map images to points in hyper-space Similarity is based on distance measure Similarity is based on distance measure

4. Relevance Feedback Based on Parameter Estimation of Target Distribution Feature Extraction Model R B G

5. Relevance Feedback Based on Parameter Estimation of Target Distribution Relevance Feedback Relevance feedback Relevance feedback Architecture to capture user’s target of search Architecture to capture user’s target of search Learning process Learning process Two steps Two steps Feedback – how to learn from the user’s relevance feedback Feedback – how to learn from the user’s relevance feedback Display – how to select the next set of documents and present to user Display – how to select the next set of documents and present to user

6. Relevance Feedback Based on Parameter Estimation of Target Distribution 1 st iteration User Feedback Display 2 nd iteration Display User Feedback Estimation & Display selection Feedback to system

7. Relevance Feedback Based on Parameter Estimation of Target Distribution Former Approaches Multimedia Analysis and Retrieval System (MARS) Multimedia Analysis and Retrieval System (MARS) Yong Rui et al. Relevance feedback: A powerful tool for interactive content-based image retrieval. - 1998 Yong Rui et al. Relevance feedback: A powerful tool for interactive content-based image retrieval. - 1998 Using weight to capture user’s preference Using weight to capture user’s preference Pic-Hunter Pic-Hunter Ingemar J. Cox et al. The Bayesian image retrieval system, pichunter, theory, implementation, and psychophysical experiments. - 2000 Ingemar J. Cox et al. The Bayesian image retrieval system, pichunter, theory, implementation, and psychophysical experiments. - 2000 Images are associated with a probability being the user’s target Images are associated with a probability being the user’s target Bayesian learning Bayesian learning

8. Relevance Feedback Based on Parameter Estimation of Target Distribution Comparison MARSPic-Hunter Our approach Capturing user’s target of search Weight on different feature and dimension Probability associated with images Estimated parameter of target cluster Updating Counting and variance Bayes’ rule EM algorithm Display Most likely Maximum Entropy Principle

9. Relevance Feedback Based on Parameter Estimation of Target Distribution The Model Feature Extraction Feature Extraction I - raw image data I - raw image data  - set of feature extraction method  - set of feature extraction method f - feature extraction operation f - feature extraction operation Images  data point in hyper-space (R d ) Images  data point in hyper-space (R d ) Problem scope is narrowed down to a particular feature Problem scope is narrowed down to a particular feature

10. Feedback

11. Relevance Feedback Based on Parameter Estimation of Target Distribution Inconsistence in Feedback User tells lies User tells lies Too many false positive or false negative Too many false positive or false negative Conflict of feedback in each iteration by careless mistake Conflict of feedback in each iteration by careless mistake

12. Relevance Feedback Based on Parameter Estimation of Target Distribution Resolving Conflicts How to deal with inconsistent user feedback? How to deal with inconsistent user feedback? Maintain a relevance measure for each data points Maintain a relevance measure for each data points Relevance measure > 0 counted as relevant and use in estimation Relevance measure > 0 counted as relevant and use in estimation

13. Relevance Feedback Based on Parameter Estimation of Target Distribution Estimating Target Distribution User’s target is a cluster User’s target is a cluster Assume it follows a Gaussian distribution Assume it follows a Gaussian distribution Model a distribution that fits the relevant data points Model a distribution that fits the relevant data points Based on the parameter of distribution, system learns what user wants Based on the parameter of distribution, system learns what user wants Data points selected as relevant Red

14. Relevance Feedback Based on Parameter Estimation of Target Distribution Expectation Maximization Fitting a Gaussian distribution function using feedback data points Fitting a Gaussian distribution function using feedback data points By expectation maximization By expectation maximization Distribution represent user’s target Distribution represent user’s target Expectation function match the display model Expectation function match the display model

15. Relevance Feedback Based on Parameter Estimation of Target Distribution Updating Parameters Estimated mean is the average Estimated mean is the average Estimated variance by differentiation Estimated variance by differentiation Iterative approach Iterative approach

16. Display

17. Relevance Feedback Based on Parameter Estimation of Target Distribution Maximum Entropy Display Why maximum entropy display? Why maximum entropy display? Reason: fully utilize information contained in user feedback to reduce number of feedback iteration Reason: fully utilize information contained in user feedback to reduce number of feedback iteration Result: near boundary images will be selected to fine tune parameters Result: near boundary images will be selected to fine tune parameters

18. Relevance Feedback Based on Parameter Estimation of Target Distribution Maximum Entropy Display How to simulate maximum entropy display in our model? How to simulate maximum entropy display in our model? Data points 1.18  away from  are selected Data points 1.18  away from  are selected Why 1.18? Why 1.18? 2P(  +1.18  )=P(  ) 2P(  +1.18  )=P(  ) Query target cluster center Selected by knn search Selected by Max. Entropy

19. Relevance Feedback Based on Parameter Estimation of Target Distribution Experiment Synthetic data generated by Matlab Synthetic data generated by Matlab Mixture of Gaussians Mixture of Gaussians Class label of data points shown for reference to give feedback Class label of data points shown for reference to give feedback Dose it works and works better? Dose it works and works better?

20. Relevance Feedback Based on Parameter Estimation of Target Distribution Convergence Is the estimated parameter (mean and variance) converge to the actual parameter of target distribution? Is the estimated parameter (mean and variance) converge to the actual parameter of target distribution? Is the maximum entropy display correctly done? Is the maximum entropy display correctly done?

21. Relevance Feedback Based on Parameter Estimation of Target Distribution

24. Performance Compares to Rui’s intra-weight updating model Compares to Rui’s intra-weight updating model Nearest neighbour search performed after several feedbacks (6-7 iterations) Nearest neighbour search performed after several feedbacks (6-7 iterations) Data points outside 2  are discarded in our algorithm Data points outside 2  are discarded in our algorithm Precision-Recall graph Precision-Recall graph

25. Relevance Feedback Based on Parameter Estimation of Target Distribution

31. Future Works Modification to learn from information contained in non-relevant set Modification to learn from information contained in non-relevant set To capture correlation in different features To capture correlation in different features Apply in CBIR system for performance measurement Apply in CBIR system for performance measurement

32. Relevance Feedback Based on Parameter Estimation of Target Distribution Conclusion Proposed an approach to interpret the feedback information from user and learn his target of search Proposed an approach to interpret the feedback information from user and learn his target of search Compares our approach with Rui’s intra- weight updating method Compares our approach with Rui’s intra- weight updating method

33. END Presentation file available at http://www.cse.cuhk.edu.hk/~kcsia/research/