1. 1 Content Based Image Retrieval Using MPEG-7 Dominant Color Descriptor Student: Mr. Ka-Man Wong Supervisor: Dr. Lai-Man Po MPhil Examination Department of Electronic Engineering City University of Hong Kong August 2004

2. 2 Outlines of this presentation Objectives MPEG-7 visual descriptors A new similarity measure for dominant color descriptor Merged Palette Histogram Similarity Measure A new relevance feedback for dominant color descriptor Merged Palette Histogram Relevance Feedback MIRROR – A CBIR system using MPEG-7 visual descriptors Conclusions

3. 3 Objective of this research study To investigate Content Based Image Retrieval (CBIR) based on color features To develop efficient techniques for MPEG-7 Dominant Color Descriptor (DCD) Merged Palette Histogram Similarity Measure Merged Palette Histogram Relevance Feedback Apply proposed methods into a real system

4. 4 MPEG-7 visual descriptors Color Color structure, scalable color, dominant color, color layout Texture Homogeneous texture, edge histogram, texture browsing Shape Contour shape, region shape, 3D shape Motion (for video contents) Motion activity, camera motion, motion trajectory, parametric motion They describe image/video contents in different aspects

5. 5 MPEG-7 color descriptors Dominant color descriptor (DCD) A compact color descriptor generated by color quantization Color structure descriptor (CSD) Color histogram generated by structure block scanning approach Scalable color descriptor (SCD) Color histogram in a quantized HSV space with Haar transform. Color layout descriptor (CLD) A compact color-spatial descriptor generated by dividing the image by a 8x8 gird with DCT transform.

6. 6 Relevance feedback Color might perform well, but it might not match user ’ s expectation Effectiveness could be further improved by involving users in the searching

7. 7 MPEG-7 color descriptors Two major problems are found in DCD make it unable to perform well Problems of its original similarity measure method Cannot use relevance feedback easily We will focus on DCD in this research study New methods are developed to utilize DCD Similarity measure Relevance feedback

8. 8 Merged palette histogram similarity measure for dominant color descriptor Dominant Color Descriptor Shortcomings of the existing similarity function Proposed Merged Palette Histogram Similarity Measure

9. 9 Dominant Color Descriptor Feature representation The dominant colors Percentage of area of the dominant color Maximum of 8 colors Dominant Color Descriptor percentage color

10. 10 Dominant Color Descriptor Feature extraction GLA color quantization Each color have at least T d distance away in a perceptually uniform CIELuv Dominant Color Descriptor Original Image percentage color CQ Color Quantized Image

11. 11 Dominant Color Descriptor Similarity measure A modified Quadratic Histogram Distance Measure (QHDM) Percentage p color Percentage q color

12. 12 Dominant Color Descriptor Since each DCD may have different set of colors, QHDM is used to account for identical colors and similar colors. Percentage p color Percentage q color

13. 13 Shortcomings of the QHDM similarity function Limitations of QHDM Distance upper bound is not fixed Completely different image cannot be identified by its upper bound The similarity coefficient does not well model color similarity It does not balance between color distance and area of matching The new Merged Palette Histogram Similarity Measure method Can compare identical colors as well as similar colors Use area of matching for similarity measure

14. 14 Proposed Merged Palette Histogram Similarity Measure MPHSM Process - 1 Find the closest pair of colors using Euclidian distance in CIELuv color space

15. 15 Proposed Merged Palette Histogram Similarity Measure MPHSM process - 2 If the distance smaller than a threshold T d, merge them to form a new common palette color Step 1 – 2 iterates until the minimum distance larger than T d

16. 16 Proposed Merged Palette Histogram Similarity Measure MPHSM process - 3 A new common palette is then generated Form new descriptors based on the common palette Dominant Color Descriptor Common Palette Merged Palette Histogram

17. 17 Proposed Merged Palette Histogram Similarity Measure MPHSM process - 4 Histogram intersection is used to measure the similarity Count the non-overlapping area as the distance

18. 18 Experimental results MPHSM improves DCD for both datasets While using Corel_1k dataset MPHSM outperforms QHDM significantly *ANMRR (smaller means better) ANMRR (MPEG-7 CCD)ANMRR (Corel_1k) DCD-MPHSM0.26040.3946 DCD-QHDM0.28340.5648

19. 19 Experimental results Visual results - Query #32 from MPEG-7 CCD Query image QHDM results, ANMRR=0.4MPHSM result, ANMRR=0.0111

20. 20 Experimental results Visual results – Query #15 from Corel_1k Query image QHDM result, ANMRR=0.6464MPHSM result, ANMRR=0.4819

21. 21 Conclusions on Merged Palette Histogram Similarity Measure MPHSM generates a common palette Can match similar colors Uses area of matching as the similarity Boosts DCD in terms of ANMRR Gives better visual results

22. 22 Merged palette histogram for dominant color descriptor relevance feedback Feature weighting relevance feedback technique and its limitations Proposed Merged Palette Histogram Relevance Feedback Experimental results

23. 23 Feature weighting relevance feedback technique and its limitations Feature weighting relevance feedback technique Assumes a fixed feature space (histograms) Taking liner combinations on matching histogram bins. Simple approach: Histogram averaging +() / 2 =

24. 24 Feature weighting relevance feedback technique and its limitations But DCDs of images might have different set of colors, similar images might not have any exactly matched colors. Two problems H1H1 H2H2 H’

25. 25 Limitation of feature weighting relevance feedback technique Problems The number of colors in updated query may greatly exceed the limit of the number of colors defined by MPEG-7 as the number of selected images increase. Similar colors are separated. By definition of DCD, similar colors should be grouped together. H1H1 H2H2 H’

26. 26 Limitation of feature weighting relevance feedback technique The Merged Palette Histogram Relevance Feedback The updated query contains common colors among selected images Represent the selected images efficiently

27. 27 Proposed Merged Palette Histogram for Relevance Feedback Merged Palette Histogram Relevance Feedback (MPH-RF) process - initialize Obtain all DCD from selected images

28. 28 Proposed Merged Palette Histogram for Relevance Feedback Merged Palette Histogram Relevance Feedback (MPH-RF) process - 1 Link all DCD together ++ = 6 colors 8 colors6 colors20 colors

29. 29 Proposed Merged Palette Histogram for Relevance Feedback Merged Palette Histogram Relevance Feedback (MPH-RF) process - 2 Palette Merging Find the closest pair of colors based on Euclidian distance in CIELuv If minimum distance smaller than T d merge the color pair and sum up the percentages of merged colors Iterate until minimum distance > T d 20 colors 9 colors

30. 30 Proposed Merged Palette Histogram for Relevance Feedback Merged Palette Histogram Relevance Feedback (MPH-RF) process - 3 Approximation Cut the least significant colors if number of colors >8 9 colors8 colors

31. 31 Proposed Merged Palette Histogram for Relevance Feedback Merged Palette Histogram Relevance Feedback (MPH-RF) process - 4 Re-normalization Adjust the histogram sum into 1 An updated query is generated Approximated MPHUpdated Query Histogram Sum =1

32. 32 Experimental results MPH-RF gives improvement on all combinations of similarity measures and datasets. Combination of MPHSM and MPH-RF gives a significant improvement Three iterations of relevance feedback give a significant result *ANMRR – smaller means better MPEG-7 CCDCorel_1k InitialAfter 3 RF RF Improvement InitialAfter 3 RF RF Improvement DCD- MPHSM 0.26040.17520.08520.39460.32980.0648 DCD- QHDM 0.28340.21170.07170.54680.49000.0568

33. 33 Experimental results Visual results – Query #50 from MPEG-7 CCD, MPHSM Query image Ground truth images Initial retrieval, 4 of 8 ground truths hit, NMRR=0.5 First RF retrieval, 6 of 8 ground truths hit, NMRR=0.2782 Second RF retrieval, 7 of 8 ground truths hit, NMRR=0.1541

34. 34 Experimental results Visual results – Query #13 from Corel_1k, MPHSM Query image Ground truth images Initial retrieval, 7 of 11 ground truths hit, NMRR=0.3043 First RF retrieval, 9 of 11 ground truths hit, NMRR=0.1688 Second RF retrieval, 9 of 11 ground truths hit, NMRR=0.1688

35. 35 Conclusions on Merged Palette Histogram Relevance Feedback MPH-RF generates a new DCD query using palette merging technique Represents the selected relevant images naturally and effectively MPH-RF boosts all situations of DCD searching

36. 36 MIRROR – A CBIR system using MPEG-7 visual descriptors MPEG-7 Image Retrieval Refinement based On Relevance feedback Systems structure Demo

37. 37 MIRROR – A CBIR system using MPEG-7 visual descriptors System structure Image DB Similarity Measure Relevance Feedback user initial inputuser feedback MPEG-7 data Feature Extraction reference imagerelevantimage(s) Similarity Sorting Output Images user feedback -7 )

38. 38 MIRROR – A CBIR system using MPEG-7 visual descriptors Demo Demo 1: Similarity Measure Demo 2: Relevance Feedback http://www.ee.cityu.edu.hk/~mirror/

39. 39 Conclusions of this research work By utilizing MPHSM and MPH-RF DCD, DCD becomes compact as well as accurate Similarity measure Merged Palette Histogram Similarity Measure Relevance Feedback Merged Palette Histogram Relevance Feedback Proposed methods are implemented into a real system. CBIR functions Evaluation tools

40. 40 Q & A