1. Content-based Image Retrieval Hai Le Supervisor: Sid Ray

2. Content-based Image Retrieval ► Introduction ► CBIR fundamentals ► Overview of the system ► Relevance feedback ► Results and Discussion ► Conclusion

3. Content-based Image RetrievalIntroduction ► Search and retrieval of image databases ► Retrieval by query, such as an example image ► Application areas include, weather forecasting, iiiiiiiimedical research, fabric design, WWW search iiiiiiiijust to name a few.. ► CBIR systems which are commercially iiiiiiiiavailable include IBM’s QBIC, Blobworld, iiiiiiiiVisualSeek, Virage etc.

4. Content-based Image RetrievalCBIR Fundamentals Input query and image database Extract Image features Index images by sorting the images according to a distance measurement Determine feature weighting Present retrieved images to user User Feedback? N Y

5. Content-based Image RetrievalCBIR Fundamentals Input query and image database Extract Image features Index images by sorting the images according to a distance measurement Determine feature weighting Present retrieved images to user User Feedback? N Y

6. Content-based Image RetrievalCBIR Fundamentals ► User is searching for a specific target item from iiiiiiiiia known database ► User is searching for a class of similar items ► Query by example image, drawing etc. ► Exploiting multiple queries to refine searches ► Use of segmentation or object selection to iiiiiiiiisearch for image specifics

7. Content-based Image RetrievalCBIR Fundamentals Input query and image database Extract Image features Index images by sorting the images according to a distance measurement Determine feature weighting Present retrieved images to user User Feedback? N Y

8. Content-based Image RetrievalCBIR Fundamentals ► Low level image features used as classifiers ► Most commonly used features include: ► Colour ► Texture ► Shape ► Transform domain features ► DCT ► Wavelet filters

9. Content-based Image RetrievalCBIR Fundamentals Input query and image database Extract Image features Index images by sorting the images according to a distance measurement Determine feature weighting Present retrieved images to user User Feedback? N Y

10. Content-based Image RetrievalCBIR Fundamentals ► Features are represented as a numerical value ► Similarities between images are determined by iiiiiiiiidifferences between values ► Typically, queries are viewed as points in a iiiiiiiiimultidimensional feature space, and the iiiiiiiii‘distance’ between points is determined ► Common distance functions include ► Euclidean ► City block ► Images are indexed in terms of their distance iiiiiiiiifrom the query point

11. Content-based Image RetrievalCBIR Fundamentals Input query and image database Extract Image features Index images by sorting the images according to a distance measurement Determine feature weighting Present retrieved images to user User Feedback? N Y

12. Content-based Image RetrievalCBIR Fundamentals ► Top N most similar images are retrieved ► Searches can be refined by user feedback ► Data which is fed back is usually used to iiiiiiiidetermine significant features ► A feature’s influence on a query can be iiiiiiiiincreased/decreased by applying a weighting iiiiiiiifacture during distance calculations ► Images are re-indexed

13. Content-based Image RetrievalOverview of the System ► Query by example image ► 3 selectable features for querying, with 189 iiiiiiiisub features ► Selection between 4 weighting schemes ► Incorporates user interaction, retrieved images iiiiiiiiare selected as either relevant or non-relevant

14. Content-based Image RetrievalOverview of the System ► Features used in the system: ► Colour histogram ► Colour moments ► Edge histogram

15. Content-based Image RetrievalOverview of the System ► Colour histogram using linear quantised HSV iiiiiiiiicolour space ► 18 hues (20 degrees separation), 3 iiiiiiiiisaturations, 3 values, total of 162 colours ► Each bin is proportional ie. ► Colour moments calculated by taking the iiiiiiiiiaverage, standard deviation and cube root of iiiiiiiiithe third moment, of each of the HSV channels

16. Content-based Image RetrievalOverview of the System ► Edge histogram is derived by the number of iiiiiiiiedge pixels present in the image and the iiiiiiiidirection of the edge pixels ► Sobel edge detector is applied on the image, if iiiiiiiithe edge strength surpasses a threshold the iiiiiiiidirection is recorded and quantised ► A histogram is generated from all the iiiiiiiidirectionality values ► Each bin represents a sub feature

17. Content-based Image RetrievalOverview of the System ► Euclidean metric used to measure distance ► Features which constitute large values iiiiiiiiovershadow those with small values ► Gaussian normalisation is applied to give equal iiiiiiiiemphasis on all features

18. Content-based Image RetrievalOverview of the System ► Most similar images are those closest to the iiiiiiiiquery image in the feature space ► Images are indexed by their distance values ► The top N images are retrieved and displayed

19. Content-based Image RetrievalRelevance Feedback ► An image object can be modeled as: ► D is the image data, F = { f i }, features, R = { r ij } iiiiiiiia set of representation or subfeatures, iiiiiiiieg. Histogram bins ► An objects similarity to the query is calculated iiiiiiiiby r ij and its corresponding weight w ij and a iiiiiiiisimilarity measure S ► Weighted Euclidean:

20. Content-based Image RetrievalRelevance Feedback ► Different queries are more reliant on certain iiiiiiiiifeatures than others ► Varying w ij ’s changes emphasis of features iiiiiiiiiduring distance calculation ► Term weighting using relevance feedback iiiiiiiiiinclude: ► Density estimation ► Support vector machine learning ► Self-organising maps

21. Content-based Image RetrievalRelevance Feedback ► System training data uses binary feedback, iiiiiiiiieither positive example or negative examples ► Number of retrieved images is 4 plus original iiiiiiiiiquery image ► Of the 4 examples, images marked as relevant iiiiiiiiiare placed in a positive subset, unselected iiiiiiiiiimages are placed in a negative subset ► Unretrieved images are left indifferent

22. Content-based Image RetrievalRelevance Feedback ► Weighting scheme devised by Rui et. al ► Form an M x N matrix, M = number of positive iiiiiiiiimages, N = number of features ► For each column of the matrix calculate the iiiiiiiistandard deviation ► The weight w ij is calculated by taking:

23. Content-based Image RetrievalRelevance Feedback ► Using both positive and negative images ► If means of r ij positive and r ij negative are iiiiiiiiisimilar, feature is insignificant ► Using variance:

24. Content-based Image RetrievalRelevance Feedback ► Positive images are similar because of a iiiiiiiiispecific characteristic, negative images are iiiiiiiiidifferent because of a number of iiiiiiiiicharacteristics ► Using negative standard deviation

25. Content-based Image RetrievalResults and Discussion ► Image database of 200, divided into individual iiiiiiiigroups ranging from 4-10 images per group ► group of 10 images selected for test case ► each image selected as query, precision and iiiiiiiiirecall calculated for each of the 4 term iiiiiiiiweighting schemes after 1 iteration ► Precision and recall averaged over the 10 iiiiiiiiimages

26. Content-based Image RetrievalResults and Discussion

27. Content-based Image RetrievalResults and Discussion

28. Content-based Image RetrievalResults and Discussion ► Employing negative examples, showed iiiiiiiiiimprovement ► Type 3 showed slowest degradation ► Type 4 showed significant improvement

29. Content-based Image RetrievalConclusion ► CBIR system developed incorporating ► variety of features ► functional interface ► ‘user friendly’ feedback mechanism ► Improved term weighting scheme over used iiiiiiiiin the MARS CBIR system ► Maintains same simplistic interface as MARS ► Future work ► image classification ► segmentation/local features ► better feature representation eg. Gabor filtering