1. Relevance Feedback and User Interaction for CBIR Hai Le Supervisor: Dr. Sid Ray

2. Outline Introduction to CBIR CBIR query structure Feature weighting Objectives Plans and approaches Current progress Conclusion

3. Introduction to CBIR CBIR systems are required to meet users needs for image retrieval Retrieval by query, such as an example image Application areas include, weather forecasting, medical research, fabric design, WWW search just to name a few.. CBIR systems which are commercially available include IBM’s QBIC, Blobworld, VisualSeek, Virage etc.

4. Current Approaches to CBIR Current approaches follow a similar routine Extract lower level features of the images Measure the degree of similarity between image features Apply weighting to the most predominate features Index the images based on the combined similarity of features

5. Image Features Colour – Global colour histogram – Colour correllogram Texture – Coarseness – Contrast – Directionality Shape – Area – Shape moments

6. Indexing and Similarity Measurements Queries are translated into a point in a multi dimensional feature space The most similar images are those which are closest to the query point, numerical values can be obtained Common distance measures include: – Euclidean – City block Some indexing structures include: – Bayesian networks

7. Feature Weighting and Relevance Feedback Many features may be used to determine similarity These features need to be combined Some features may be deemed more important than others Feature weighting provides a means to distinguish the importance of various features Questions which arise include, how do we determine the importance of particular features?

8. Summary of CBIR query structure Input query image and image database Extract Image features Index images by sorting the images according to a distance measurement Calculate the weighting of the various features Present retrieved images to user User Feedback?

9. Summary of CBIR query structure Extract Image features Index images by sorting the images according to a distance measurement Calculate the weighting of the various features Present retrieved images to user User Feedback? Input query image and image database

10. Feature Weight Calculations Assign higher weights to more important features and lower weights to less importance features Deciding which features are more important: – Analyse the contents of the image database and determine weights based on the variation of features – Use user perception and relevance feedback to determine which features are more important

11. Methods of weight calculation Analyse the feature content of the images in the database in respect to the features of the query image Features with high amounts of variation are deemed more important than those with small amounts of variation These features can be given a higher weighting during distance calculations

12. Methods of Weight calculation Hore and Ray methods involved – Using the standard deviation (σ) of the distances between the query image and the database images for a particular feature. A high standard deviation would mean a large amount of variation for that particular feature – Using the standard deviation of the entire image database divided by the standard deviation of the top N images for a particular feature. Second method provided the best results

13. Methods of Weight calculation Use of user perception to determine the importance of various features User selects positive and negative images from the retrieved image set Different systems handle the positive and negative images in different ways some of these include: – Determining the spread of the positive and negative images – Comparing the positive and negative images to the original query image

14. Methods of Weight calculation Naster et. al, proposed that the probabilities of the images in the database are updated depending on the location of positive and negative images MUSE developed by Marques et. al uses a unique system in which images which are similar for a certain features are clustered together The images which belong to certain clusters are updated depending on the positive and negative images selected by the user Feature weighting is calculated depending on the users selection habits

15. Objectives of the project Devise a feature weighting scheme based on the approaches previously mentioned Devise an analysis for feature weighting Develop a CBIR testbed

16. Plans and approaches Develop a simple CBIR testbed using colour and shape features Build on the weighting scheme devised by Hore and Ray by including user interaction Use rank correlation coefficient to distinguish between computational and perceptual results

17. Current progress Image database of 40 images including: – 6 groups of 5 images containing similar images based on colour and shape – 1 group of 10 images containing random images Simple CBIR testbed using histogram matching based on a quantised RGB colour space Used R, G, B channels as separate features, for testing purposes Weight calculation based Hore and Ray method using standard deviation

18. Conclusion CBIR systems have numerous applications, and incorporate many areas of computing There is a discrepancy between computational results and perceptual results due to the use of low level image features These problems can be overcome by using feature weighting and by involving the “user in the loop” The project will build on previous feature weighting methods

19. More about the project http://csse.monash.edu.au/~hle hle@csse.monash.edu.au