1. Images Similarity by Relative Dynamic Programming M. Sc. thesis by Ady Ecker Supervisor: prof. Shimon Ullman

2. Overview The similarity problem The Method: Overlapping patches Combining evidences Relative dynamic programming Analysis

4. The similarity problem Old problem. Important. Difficult. Natural to humans.

5. What is similarity? Similarity has many aspects, e.g. shape, color, size, outline, texture. Relative and context dependent. Depends on experience with classes in the real world.

6. Similarity is not a metric Triangle inequality Symmetry Minimality

7. Previous approaches to similarity computation Distance functions (L 2, correlation). Features detection, image signatures. Minimal energy aligning transformation. Dynamic programming measures. Overlapping fragments (Ullman and Sali).

8. Motivation Point-wise methods fail on very simple examples.

9. Motivation We propose: common substructures of multiple sizes at multiple scales.                                  

10. Similarity of sub-patches at the local level A1A1 B1B1 C1C1 A2A2 C2C2 B2B2 A1A1 A2A2 B1B1 B2B2 C1C1 C2C2

12. Overlapping sub-patches overlapping fragments impose similar structure.

13. Hierarchical decomposition Allows the sub-patches to move a little.

14. Hierarchical alignment

15. Similarity indicators Each indicator has its marginal distribution, but the joint distribution is too complex. The indicators are dependent. FXFX X normalization by 1-D distribution

16. Generalization groups example The similarity is related to the intersection of the groups: Pr(    (A,B)  length ratio  ratio(A,B) )   45º length ratio  2

17. Graphical illustration of the similarity score independent case general case estimation error z F 01 0 1 z (x,y)(x,y) area z prob. F F z 01 z (x0,y0)(x0,y0) (x1,y1)(x1,y1) z z 1 0 (x3,y3)(x3,y3) F 01 0 1 (x0,y0)(x0,y0) (x1,y1)(x1,y1) F F F

18. Empirical example 10,000 points from 2D normal distribution. Normalization: x → Pr ( X  x ) y → Pr ( Y  y ) Only 233 points between the two curves.

19. The similarity score The score combines several similarity indicators. The score is relative. Since the score is normalized, it can be used in hierarchical construction: x1x1 x2x2 x3x3 y1y1 y2y2 z

20. Dynamic programming in 2D? Vertical movement Horizontal and vertical movements

21. Relative Dynamic Programming (RDP) Works in 2D like dynamic programming algorithms, but: Scores are relative. No explicit objective function. Start from many points simultaneously.

22. I1I1 I2I2 D(x,x  2) D(x,x  3) D(x,x  2) D(x,x  1) D(x,x  3) D(x,x)D(x,x) D(x,x  1) T(i,)T(i,) 3 2 1 0 11 22 33 xx The similarity table

23. The minima table Allow flexible movements. Costs keep movement coherence. Used for efficiency. Bi-directional (two tables). projected index delta

24. Combining evidences

25. Flowchart Patch size = N? Input images Set patch size = 2 Basic similarity measure Store scores in similarity table Yes Output the similarity score Build minima tables Normalization Double the patch size Geometric mean of sub-patches' scores No

26. Weighting patches

27. Multi-scale Two dimensions, cross-resolution: T(r,z,i 1,i 2,  1,  2 )=min(P, V high-low, V low-low ) P V low-low V high-low low resolution: high resolution:

28. Distributions Distributions are sampled empirically. The sampling is done layer by layer. distribution of the geometric mean of independent uniform variables distribution of the weighted geometric mean of real images

30. Performance The complexity is O(N 2  2 ). The number of operations is governed by the operations on the original resolution. Basic-similarity operations: 32  32  17  17  16  4,735,000. Minima operations: 32  32  17  17  9  2  5,327,000. Total  10,000,000. Takes a second on 1.5GH p.c.

31. Results RDP L2L2 correlation

32. Results

33. Experimental conclusions The algorithm is superior to point-wise distance functions (L 2, correlation), even when the images are aligned by a global transformation. The current implementation neglects important parts such as outlines and topology.

34. Qualitative properties Similarity by parts. Captures variety of transformations. Weights the salient parts. Adaptive to distributions in the domain.

35. Properties of the design Robust. Simple operations. Can be implemented in parallel and neural-networks. Intuitive. Extendible to deal with many aspects of similarity.

36. Summary We presented a new scheme to the similarity problem in general. The scheme is based on the systematic evaluation of similarity on overlapping sub-patches. Scores are relative. Scores are combined hierarchically using empiric normalizations. The implementation captures aspects of perceptual similarity, but is still inferior to human ’ s judgment.