Images Similarity by Relative Dynamic Programming M. Sc. thesis by Ady Ecker Supervisor: prof. Shimon Ullman

Overview The similarity problem The Method: Overlapping patches Combining evidences Relative dynamic programming Analysis

The similarity problem Old problem. Important. Difficult. Natural to humans.

What is similarity? Similarity has many aspects, e.g. shape, color, size, outline, texture. Relative and context dependent. Depends on experience with classes in the real world.

Similarity is not a metric Triangle inequality Symmetry Minimality

Previous approaches to similarity computation Distance functions (L 2, correlation). Features detection, image signatures. Minimal energy aligning transformation. Dynamic programming measures. Overlapping fragments (Ullman and Sali).

Motivation Point-wise methods fail on very simple examples.

Motivation We propose: common substructures of multiple sizes at multiple scales.                                  

Similarity of sub-patches at the local level A1A1 B1B1 C1C1 A2A2 C2C2 B2B2 A1A1 A2A2 B1B1 B2B2 C1C1 C2C2

Overlapping sub-patches overlapping fragments impose similar structure.

Hierarchical decomposition Allows the sub-patches to move a little.

Hierarchical alignment

Similarity indicators Each indicator has its marginal distribution, but the joint distribution is too complex. The indicators are dependent. FXFX X normalization by 1-D distribution

Generalization groups example The similarity is related to the intersection of the groups: Pr(    (A,B)  length ratio  ratio(A,B) )   45º length ratio  2

Graphical illustration of the similarity score independent case general case estimation error z F z (x,y)(x,y) area z prob. F F z 01 z (x0,y0)(x0,y0) (x1,y1)(x1,y1) z z 1 0 (x3,y3)(x3,y3) F (x0,y0)(x0,y0) (x1,y1)(x1,y1) F F F

Empirical example 10,000 points from 2D normal distribution. Normalization: x → Pr ( X  x ) y → Pr ( Y  y ) Only 233 points between the two curves.

The similarity score The score combines several similarity indicators. The score is relative. Since the score is normalized, it can be used in hierarchical construction: x1x1 x2x2 x3x3 y1y1 y2y2 z

Dynamic programming in 2D? Vertical movement Horizontal and vertical movements

Relative Dynamic Programming (RDP) Works in 2D like dynamic programming algorithms, but: Scores are relative. No explicit objective function. Start from many points simultaneously.

I1I1 I2I2 D(x,x  2) D(x,x  3) D(x,x  2) D(x,x  1) D(x,x  3) D(x,x)D(x,x) D(x,x  1) T(i,)T(i,) 11 22 33 xx The similarity table

The minima table Allow flexible movements. Costs keep movement coherence. Used for efficiency. Bi-directional (two tables). projected index delta

Combining evidences

Flowchart Patch size = N? Input images Set patch size = 2 Basic similarity measure Store scores in similarity table Yes Output the similarity score Build minima tables Normalization Double the patch size Geometric mean of sub-patches' scores No

Weighting patches

Multi-scale Two dimensions, cross-resolution: T(r,z,i 1,i 2,  1,  2 )=min(P, V high-low, V low-low ) P V low-low V high-low low resolution: high resolution:

Distributions Distributions are sampled empirically. The sampling is done layer by layer. distribution of the geometric mean of independent uniform variables distribution of the weighted geometric mean of real images

Performance The complexity is O(N 2  2 ). The number of operations is governed by the operations on the original resolution. Basic-similarity operations: 32  32  17  17  16  4,735,000. Minima operations: 32  32  17  17  9  2  5,327,000. Total  10,000,000. Takes a second on 1.5GH p.c.

Results RDP L2L2 correlation

Results

Experimental conclusions The algorithm is superior to point-wise distance functions (L 2, correlation), even when the images are aligned by a global transformation. The current implementation neglects important parts such as outlines and topology.

Qualitative properties Similarity by parts. Captures variety of transformations. Weights the salient parts. Adaptive to distributions in the domain.

Properties of the design Robust. Simple operations. Can be implemented in parallel and neural-networks. Intuitive. Extendible to deal with many aspects of similarity.

Summary We presented a new scheme to the similarity problem in general. The scheme is based on the systematic evaluation of similarity on overlapping sub-patches. Scores are relative. Scores are combined hierarchically using empiric normalizations. The implementation captures aspects of perceptual similarity, but is still inferior to human ’ s judgment.