1 Scale and Rotation Invariant Matching Using Linearly Augmented Tree Hao Jiang Boston College Tai-peng Tian, Stan Sclaroff Boston University

Scale and Rotation Invariant Matching 2

Previous Methods  Hough Transform (Duda & Hart) and RANSAC (Fischler and Bolles)  Dynamic programming (Felzenszwalb and Huttenlocher 05)  Loopy belief propagation (Weiss and Freeman 01)  Tree-reweighted message passing (Kolmogorov 06)  Primal-dual methods (Komodakis and Tziritas 07)  Dual decomposition (Komodakis, Paragios and Tziritas 11, Torresani, Kolmogorov and Rother 08)  Successive convexification (Jiang 2009, Li, Kim, Huang and He 2010) 3

Unsolved Issue  How to find the optimal rotation angle and scale especially if the ranges are unknown? 4 Quantizing rotation angle and scale

The Model We all Want to Have 5

6 In Reality We Need to Use … Hyperedge Non-tree edge

Linearly Augmented Tree (LAT) 7 Any tree constraints Linear non-tree constraints LAT works on continuous scale and rotation and non-tree structure.

Optimizing Invariant Matching 8 Total local feature matching cost p f(p) q f(q) cost(p,f(p)) cost(q,f(q)) … …

9 In matrix form: X X Binary assignment Matrix C C Local matching cost matrix Optimizing Invariant Matching

10 Rotation and scaling consistency Model tree edges Target Optimizing Invariant Matching

11 Y p,q Pairwise assignment matrix for site pair (p,q) s 0, u 0 = sin(θ 0 ), v 0 = cos(θ 0 ) Θ p,q S p,q Rotation angle matrixScale matrix Optimizing Invariant Matching

12 Other linear global terms such as area constraints or global affine constraints. Optimizing Invariant Matching Area scaling is

The Mixed Integer Optimization 13 g(X) Subject To: Constraints on binary matrices X, Y, and continuous variables u0, v0 and s0. Unary matching cost Rotation consistency Scaling term Other global terms

Linear Relaxation 14

Special Structure 15 Objective function “Hard” constraints X, Y Auxiliary variables Auxiliary variables Easy Ones

The Solution Space 16 Solutions feasible to the “simple” problem. Solutions feasible to “hard” constraints. Optimal solution

17 Column Generation Two initial proposals and the current best estimate. Proposals

18 Column Generation Proposals Few extreme points (proposals) can be used to obtain the solution, and they can be generated iteratively.

Decompose into Dynamic Programming 19 Create the initial trellises, and find first 2 proposals k=2 Create the initial trellises, and find first 2 proposals k=2 Find out how to update the tree Update the trees Dynamic Programming and generate new proposal (k+1) Dynamic Programming and generate new proposal (k+1) Gain > 0 Yes Done

An Example 20 Template Image

An Example 21 Template

An Example 22 Target Image

An Example 23

Another Example 24 Template

Another Example 25 Target Image

Another Example 26

Complexity Comparison 27 Direct solution Direct solution

SIFT Matching Results 28 Detection Rate This paper DPRANSACTensor (Duchenne 09) Linear (Jiang 09) Affine (Li 00) 90%63%48%5%88%8%

Match Unreliable Regions 29 Detection Rate This paper DPRANSACTensor (Duchenne 09) Linear (Jiang 09) Affine (Li 00) 90%86%20%16%34%1%

Matching Unreliable Regions 30 Detection Rate This paper DPRANSACTensor (Duchenne 09) Linear (Jiang 09) Affine (Li 00) 91%73%43%11%74%14%

More Tests 31 RateTim e 90%0.78s 91%0.42s 98%0.03s 90%0.02s 94%0.07s 91%0.05s

Test on Ground Truth Data 32

Summary  LAT method can incorporate global constraints and can be efficiently solved.  It works even with very weak features and large deformations in scale and rotation invariant matching.  It works on continuous scale and rotation and does not need an upper bound for the scale.  The decomposition framework would be useful to enhance the widely applied tree methods in object detection, pose estimation and etc. 33

34 The End