1. Similarity Measures Spring 2009 Ben-Gurion University of the Negev

2. Sensor Fusion Spring 2009 Instructor Dr. H. B Mitchell email: harveymitchell@walla.co.il

3. Sensor Fusion Spring 2009 Similarity Measures Fundamental importance in all fusion Conceptually consists of two parts: Transformation T. This extracts characteristics from input image and represents it as a feature vector Distance D. This quantifies similarity of feature vectors. Symbolically:

4. Sensor Fusion Spring 2009 Similarity Measures. Example Content Based Information Retrieval (CBIR) Input: Database containing many images Requirement: Retrieve images which are similar to test image Transformation T: Represent each image in database as multi- dimensional feature vector extracted using low-level descriptors: co- occurrence matrix, color histogram, wavelets, moments etc Distance D: Euclidean, Manhattan, Minkowski, Hausdorff distances

5. Sensor Fusion Spring 2009 Similarity Measure. Shape Context In matching shapes introduced the shape context for each point on the contour C: which is a two-dimensional histogram We measure the similarity between two shapes with shapes contexts and using the test.

6. Sensor Fusion Spring 2009 Similarity Measures Metric A similarity measure is a metric if it obeys following: Experiments show that valid similarity measures generally obey the first three requirements but not the triangle inequality. Similarity measures therefore not metric

7. Sensor Fusion Spring 2009 Similarity Measures: Global vs Local Can classify similarity measures as global or local: Global measures act on the entire images and return a single scalar value Local measures act on small patches. They return a local similarity map. We can convert global measures into local measures and vice versa.

8. Sensor Fusion Spring 2009 Similarity Measures: Global vs Local Example. Global similarity measure can be used as a local measure: (m,n) Local Window W(m,n)

9. Sensor Fusion Spring 2009 Similarity Measures: Global vs Local Example. Local similarity measure can be used as a global measure: Then we aggregate the local similarity map to obtain a global measure e.g. (m,n) Local Window W(m,n)

10. Sensor Fusion Spring 2009 Global Measures Without Spatial Alignment These similarity measures are very robust but tend to have low discriminative power. Examples are the probabilistic measures:  Chernoff  Bhattacharyya  Jeffrey’s-Matusita  Kullback-Leibler  Symmetric Kullback-Leibler The measures may be applied locally. In this case typical window size is 20x20. This is needed to ensure we have sufficient pixels to calculate the probability densities p(x) and q(x).

11. Sensor Fusion Spring 2009 Distance. If we represent the distributions p(a) and q(b) as discrete histograms we may use the test. This test assumes common bins. If two images A and B have histograms Then test is Problems. (1) Optimal value for K. Rule of thumb says no bin should contain less than 5 samples. (2) No cross-bin correlations allowed

12. Sensor Fusion Spring 2009 Distance. Example The test is simple and widely used similarity measure. In shape matching we have two closed contours c and C. Each point on c is characterized by a 2D histogram and each point on C is characterized by If is associated with the point, then the match between the two histograms is We may define the overall similarity as We may also impose an order constraint on : If then we require

13. Sensor Fusion Spring 2009 Optimal Equal-Sized Histogram Bins. A recent suggestion is the following empirical formula for the optimum number of equal-sized bins in interval [0,1]: where H(l) is number of samples which fall in kth bin N is the total number of samples k is the number of equal spaced bins

14. Sensor Fusion Spring 2009 Earth Mover’s Distance. Earth Movers Distance allows cross-bin associations Given two histograms and Treat P as “supplies” where = amount of kth supply Treat Q as “demands” where = amount of lth demand EMD is defined as minimum normalized work required to transform P into Q: where is the amount transferred from to. is work involved in transferring one unit from to Project: Circular Earth Mover’s Distance Project: Pele-Werman EMD variant (A linear time histogram materic for improved SIFT matching. Pele and Werman ECCV 2008)

15. Sensor Fusion Spring 2009 Earth Mover’s Distance If the histograms P and Q are normalized then EMD is Mallows distance. If the histograms each have N bins then where the min is taken over all permutations of In one dimension this reduces to Example. Not normalized. Each box is a unit. EMD=0 Normalized. Each box is probability=0.25. For p=1, EMD=0.5

16. Sensor Fusion Spring 2009 Earth Mover’s Distance Recent development in EMD is A linear time histogram metric for improved SIFT matching by Pele and Werman ECCV 2008 Project

17. Sensor Fusion Spring 2009 Color-based Image Retrieval Jeffrey divergence Quadratic form distance Earth Mover Distanceχ 2 statistics L1 distance

18. Sensor Fusion Spring 2009 Global Measure With Spatial Alignment Tend to have higher discriminative power but require accurate registration. The measures tend to give lower similarity values as mis- registration increases. They are therefore used in spatial alignment algorithms. Simple measures are  mse  mae  Correlation coefficient Mutual Information

19. Sensor Fusion Spring 2009 Ordinal Measures The mse, mae, correlation coefficient are sensitive to illumination changes. Mutual information and other ordinal measures are designed to overcome this sensitivity by using ordered gray-levels. Ordinal measures are insensitive to illumination changes if order of the gray-levels is maintained. Two classical measures are  Spearman  Kendall Experiments on stereo matching showed Kendall measure gave best results from all ordinal measures.

20. Sensor Fusion Spring 2009 Ordinal Measures Kemeny-Snell distance comapres the realtive ranking of each ordered pair of locations in one image with its relative ranking in the second image. Smaller values of d indicate more agreement. Luo et al. suggests the measure is very powerful. However computational complexity is very high.

21. Sensor Fusion Spring 2009 Local Ordinal Measures. Bhat-Nayar We may apply the Spearman, Kendall and Kemeny-Snell distances locally. However recommendation is the local windows measure at least 20x20. Bhat-Nayar is a powerful specially designed local ordinal operator. The window size in Bhat-Nayar is from 3x3 to 13x13. Bhat-Nayar uses a local window containing K gray-levels in A and B with rank vectors Bhat-Nayar creates a composite rank vector by ranking wrt where and

22. Sensor Fusion Spring 2009 Local Ordinal Measures. Bhat-Nayar Bhat-Nayar creates a composite rank vector by ranking wrt where and Example. A: [10 20 30 50 40 70 60 90 80] B: [90 60 70 50 40 80 10 30 20] rA: [1 2 3 5 4 7 6 9 8] rB: [9 6 7 5 4 8 1 3 2] s : [9 6 7 4 5 1 8 2 3] Consider k=6.. Therefore Thus. Therefore

23. Sensor Fusion Spring 2009 Local Ordinal Measures. Bhat-Nayar Bhat-Nayar assumes all ranks are unique. How can we handle ties? Project: Use fuzzy ranks in the Bhat-Nayar scheme.

24. Sensor Fusion Spring 2009 Ordinal Measures. Mittal-Ramesh The ordinal similarity measures (spearman, kendall, Kemeny-Snell and Bhat-Nayar) are robust to changes in illumination but not to Gaussian noise. Small amounts of Gaussian noise can completely change the rankings between pixels that are not far from each other in gray- level. Mittal-Ramesh measure is an ordinal measure which also takes into accunt the pixel gray-levels. In general this should give best results. However the Mittal-Ramesh operator has very high computational complexity.

25. Sensor Fusion Spring 2009 Example Comparative results from four algorithms: Normalized Cross-Correlation Census Algorithm Bhat-Nayar Algorithm Mittal-Ramesh Algorithm

26. Sensor Fusion Spring 2009 Binary Image Measures Special measures are used for binary images Following is a local binary measure. Given two binary images A and B the corresponding distance transforms are: where Local distance map is 0 0 1 0 0 1 1 0 0 Distance transform

27. Sensor Fusion Spring 2009 Binary Image Measures May convert the local similarity map L(m,n) into a global measure. This is the Hausdorff distance: Hausdorff distance is very sensitive to noise. Robust alternatives are: