1. Supplementary Slides

2. More experimental results MPHSM already push out many irrelevant images Query image QHDM result, 4 of 36 ground truth found ANMRR=0.6464 MPHSM result, 9 of 36 ground truth found ANMRR=0.4819

3. More about experimental results Still some irrelevant image found No spatial information Cannot identify background colors Does not account for unmatched colors Initial query might not be accurate Black Background Green Background

4. More about experimental results Can be improved by Relevance Feedback Makes relevant images to have higher ranks Irrelevant normally can ’ t have higher similarity by RF But relevant images does Give more information about the interested objects Inconsistent backgrounds can be averaged out

5. More on experimental results Irrelevant images got lower rank / out of top 20 after RF Query image Ground truth images Initial retrieval, 7 of 11 ground truths hit, NMRR=0.3043 First RF retrieval, 9 of 11 ground truths hit, NMRR=0.1688 Second RF retrieval, 9 of 11 ground truths hit, NMRR=0.1688 BD EF G A E BD A BD CA E

6. More about experimental results Still some irrelevant images found Some colors are very common (Blue sky, black night, green grass, etc.) Different semantics might have similar color distribution No single feature can do perfect retrieval Can be improved by several approaches Choose suitable features Combining features

7. Suggestions on further developments For DCD Use unmatched colors Challenge 1: Did the unmatched colors representing object of interest? Or just a obstacle? Challenge 2: How to define the similarity function? Separate foreground/background Challenge 1: Can we identify it by only using DCD? Or in RF? Challenge 2: Or we need to combine other shape/texture descriptors? The DCD generation is not very accurate GLA generates an optimal for quantizing the image, it might not be accurate dominant colors. Can quantize up to 16 or more colors, and then approximate the least significant colors to obtain an 8 color DCD

8. Suggestions on further developments For general CBIR No single descriptor gives perfect retrieval Choosing suitable features Combining features (color+shape, color+texture, etc.) Automatically? Manually? How to set weights?

9. Visual description about a CBIR System flow of a CBIR system Online Process Offline Process Image DBStored Features Feature extractionUser initial inputResults outputSimilarity measure Similarity = 50% Similarity = 100% = 50% = 30%... … Feature extraction … … …

10. Color based CBIR approaches Three major approach of CBIR based on colors Area of matching – Count the area with matched colors (CSD, SCD, DCD) Color distance – Use color distance to adjust the similarity (DCD-QHDM) Spatial distribution – Matches colors having similar layout (CLD)

11. Optional parameters Spatial coherence obtained by a simple connected component analysis. A smooth surface gives a higher spatial coherence value. Color variances computed as variances of the pixel values within each cluster. But this parameter is for a dedicated similarity measure algorithm. So it is not commonly used.

12. Spatial coherency adjustment Similarity measure MPEG-7 suggests to use a modified Quadratic Histogram Distance Measure (QHDM) to measure the dissimilarity between descriptors Spatial coherency adjustment

13. Results with Spatial Coherence MPHSM improves DCD for both datasets to be more close to other non-compact descriptors While using Corel_1k dataset MPHSM outperforms CLD slightly MPHSM benefits from spatial coherency adjustment as well as QHDM DescriptorANMRR (MPEG-7 CCD)ANMRR (Corel_1k) DCD-MPHSM0.26040.3946 DCD-MPHSM with SC0.24000.3756 DCD-QHDM0.28340.5648 DCD-QHDM with SC0.24340.4958

14. DCD-QHDM upper bound problem Analysis of problem 1 The upper-bound of the distance measured varies by number of color in the descriptor Maximum of positive part is not a constant Maximum of negative part is zero So, the maximum of QHDM is not fixed This property makes DCD unable to identify completely different images by the values measured Positive partNegative part

15. Upper bound problem - example Problem 1 – The upper bound problem Consider the following images with their DCD I 1, I 2 are visually more similar than I 1, I 3 For a similarity measure that matches human perception, we can expect the distance between F 1, F 2 should be smaller than that of between F 1, F 3 F 1 F 2 1/2 F 3 1/3 I 2 I 3 I 1

16. Upper bound problem - example But distance between F 1,F 3 is smaller while measuring their distance using QHDM The extra blue color pull down the distance D 2 (F 1,F 2 )>D 2 (F 1,F 3 ) implies that I 1 is more similar to I 3 than I 2 This shows that QHDM does not meet human perception

17. DCD-QHDM Similarity coefficient problem The similarity coefficient use the color distance to fine tune the similarity Difficult to define a quantitative similarity between colors, since the sensitivity of human eye depends on many conditions (e.g. light source of the room, spatial layout of the image, etc.) 16.67% similar 44% similar T d 0% similar 1.2  d

18. Similarity coefficient problem It is easy to count 50% of area is similar. But it is difficult to count the colors are 50% similar. This method is unable to consider the area of matching and the color distance together.

19. Similarity coefficient problem - example Problem 2 – The similarity coefficient a 1i,2j problem Consider the following images I 1, I 2 are visually more similar than I 1, I 4 For a similarity measure that matches human perception, we can expect the distance between F 1, F 2 should be smaller than that of between F 1, F 4 F 4 1 I 4 F 1 F 2 1/2 I 2 I 1

20. Similarity coefficient problem - example But distance between F 1,F 4 is smaller while measuring their distance using QHDM One exactly matched color considered more important than a whole area of similar color D 2 (F 1,F 2 )>D 2 (F 1,F 4 ) implies that I 1 is more similar to I 4 than I 2 But in natural perception, images having similar color distribution is more likely to have similar semantics This shows that QHDM does not meet human perception again

21. Flow of MPHSM Initial DCDs Find a pair of colors with minimum distance d d<T d ? Merge colors having minimum distance Common Palette N Y Update each DCD based on the common palette Histogram Intersection

22. Palette Merging process, visually Example Two images with DCD, palette merging stage Dominant Color Descriptor Find the closest pair Merge colors Common Palette Merge colors Remaining colors If a remaining color is similar to any colors in the common palette. It will not included in common palette

23. About slide 23 Relationship between CBIR and Relevance Feedback (RF) The key component is query update

24. MPH-RF flow Load add DCDs Append all DCD Find closes pair of colors Minimum distance < T d ? Merge colors and percentages A A Cut least significant colors Adjust histogram sum into 1 Updated query Y N

25. RF of other MPEG-7 visual descriptors Relevance feedback for MPEG-7 descriptors Apart from the MPH-RF for DCD, we directly apply feature weighting technique on several MPEG-7 visual descriptors RF on CLD: RF on CSD:

26. RF of other MPEG-7 visual descriptors RF on SCD:

27. MIRROR – A CBIR system using MPEG-7 visual descriptors A set of visual descriptors Relevance feedback functions is added Evaluation tools MIRROR is also a development platform of MPEG-7 visual descriptors

28. Performance of color descriptors Evaluation tools Unmodified MPEG-7 reference software XM MPEG-7 Common Color Dataset (MPEG-7 CCD) with 5466 images and 50 sample queries Corel 1000 images dataset with 20 sample queries ANMRR performance metric (smaller means better) MPEG-7 CCDCorel 1000 images DescriptorANMRR Descriptor size (bytes) Descriptor size per image (bytes) ANMRR Descriptor size (bytes) Descriptor size per image (bytes) DCD0.283486,869 15.8926 (in average) 0.546818,53818.538 (in average) CLD0.225243,72880.40008,0008 CSD0.03991,401,346256.3750.3246 256,375 256.375 SCD0.1645119,56921.8750.3552119,56921.875

29. Performance of color descriptors Investigation of performances Color structure descriptor performs best among color descriptors due to its large descriptor size Dominant color descriptor performs worst, even worse than a more compact color layout descriptor “ Area of matching ” is still the most efficient approach for color based CBIR New methods will be proposed in this research to boost DCD MPEG-7 CCDCorel 1000 images DescriptorANMRR Descriptor size (bytes) Descriptor size per image (bytes) ANMRR Descriptor size (bytes) Descriptor size per image (bytes) DCD0.283486,869 15.8926 (in average) 0.546818,53818.538 (in average) CLD0.225243,72880.40008,0008 CSD0.03991,401,346256.3750.3246 256,375 256.375 SCD0.1645119,56921.8750.3552119,56921.875

30. Complete results MPHSM improves DCD for both datasets to be more close to other non-compact descriptors While using Corel_1k dataset MPHSM outperforms CLD slightly MPHSM benefits from spatial coherency adjustment as well as QHDM DescriptorANMRR (MPEG-7 CCD)ANMRR (Corel_1k) DCD-MPHSM0.26040.3946 DCD-MPHSM with SC0.24000.3756 DCD-QHDM0.28340.5648 DCD-QHDM with SC0.24340.4958 CLD0.22520.4000 CSD0.03990.3246 SCD0.16450.3552

31. Complete results MPR-RF gives significant improvement on all combinations of similarity measures and datasets. By using MPH-RF DCD can perform as good as another compact descriptor CLD, and very close to a lesser compact descriptor SCD. Three iterations of relevance feedback give a significant result Descriptor MPEG-7 CCDCorel_1k Before RFAfter 3 RF RF Improvement Before RFAfter 3 RF RF Improvement DCD- MPHSM 0.26040.17520.0852 0.39460.32980.0648 DCD-QHDM0.28340.21170.0717 0.54680.49000.0568 CLD0.2252 0.4000 CSD0.0399 0.3246 SCD0.1645 0.3552

32. Complete results The MPH-RF improvement on DCD is more significant than feature weighting for other color descriptors Color structure descriptor gives impressive results among all color descriptor, and its only drawback is the descriptor size is too large. Descriptor MPEG-7 CCDCorel_1k Before RFAfter 3 RF RF Improvement Before RFAfter 3 RF RF Improvement DCD- MPHSM 0.26040.17520.0852 0.39460.32980.0648 DCD- QHDM 0.28340.21170.0717 0.54680.49000.0568 CLD0.2252 0.18140.04380.40000.35710.0429 CSD0.0399 0.01150.02840.32460.23660.0880 SCD0.1645 0.10190.06260.35520.32760.0276