Geometry-aware Feature Matching for Structure from Motion Applications Rajvi Shah, Vanshika Srivastava, P J Narayanan Center for Visual Information Technology IIIT Hyderabad, India

Feature Matching for SfM Applications Match features across all image pairs

Pair-wise Feature Matching Image 1 + SIFTImage 2 + SIFT Img1 Descriptors Img2 Descriptors

O(M 2 ) Exhaustive Comparison Pair-wise Feature Matching Image 1 + SIFTImage 2 + SIFT Img1 Descriptors Img2 Descriptors

O(M log M) Kd-tree based Comparison Pair-wise Feature Matching Image 1 + SIFTImage 2 + SIFT Img1 Descriptors Img2 Descriptors

Features in Image 1 Features in Image 2Kd-tree of features in Image 2 (in 128-d descriptor space) Q Best Match2 nd Best Match Kd-tree based Feature Matching Search for top-two near neighbors Ratio of distances between query and 2-NNs

Features in Image 1 Features in Image 2Kd-tree of features in Image 2 (in 128-d descriptor space) Q Kd-tree based Feature Matching Best Match2 nd Best Match

Features in Image 1 Features in Image 2 Q Match F-matrix l = F.Q l Geometric Verification Distance of the matching point from epipolar line

Features in Image 1 Features in Image 2 Q F-matrix l = F.Q Distance from l > d Match l Incrorect Match d Geometric Verification Distance of the matching point from epipolar line

Main Problems  Kd-tree based approach: O(M log M)  Images with features in order on 10 4 ~  Wasted effort in matching inconsistent features  Difficult to parallelize  Ratio test is punitive for repetitive structures  Features have similar appearance, e.g. windows.  Salient but not distinctive to pass the ratio test  Rejection of many valid correspondences

Geometry-aware Matching Features in Image 1 Features in Image 2 1. Select a small subset (10%-20%) of features

Geometry-aware Matching Features in Image 1 Features in Image 2 2. Match the selected features using Kd-tree

Geometry-aware Matching Features in Image 1 Features in Image 2 3. Estimate F-Matrix using initial matches (Utilize it for geometry-guided matching) F-matrix

Features in Image 1 Features in Image 2 Q F-matrix l = F.Q Points with Distance from l < d Geometry-aware Matching 4. Find a set of candidate matches l

Features in Image 1 Features in Image 2 Q F-matrix l = F.Q Points with Distance from l < d Geometry-aware Matching 5. Find the top 2-NN within candidates & ratio test l

Features in Image 1 Features in Image 2 Q F-matrix l = F.Q Points with Distance from l < d Geometry-aware Matching Candidate set is different for each query! l

Geometry-aware Matching Efficiently finding the candidate set Distances of n points from the line l need to be computed. Linear Search: O(n)

d Geometry-aware Matching Efficiently finding the candidate set Distances of n points from the line l need to be computed. Linear Search: O(n) Select K points on line l, each at distance d. Radial Search: O(K log n) K << N.

d Geometry-aware Matching Efficiently finding the candidate set Distances of n points from the line l need to be computed. Linear Search: O(n) Select K points on line l, each at distance d. Radial Search: O(K log n) K << N.

d Geometry-aware Matching Efficiently finding the candidate set Distances of n points from the line l need to be computed. Linear Search: O(n) Select K points on line l, each at distance d. Radial Search: O(K log n) K << N. Optimize Further

d Grid based approach:  Divide the image using 4 grids of cell size 2d x 2d, overlap d G1 Geometry-aware Matching Efficiently finding the candidate set

d Grid based approach:  Divide the image using 4 grids of cell size 2d x 2d, overlap d G2 Geometry-aware Matching Efficiently finding the candidate set

d Grid based approach:  Divide the image using 4 grids of cell size 2d x 2d, overlap d G3 Geometry-aware Matching Efficiently finding the candidate set

d G4 Geometry-aware Matching Efficiently finding the candidate set Grid based approach:  Divide the image using 4 grids of cell size 2d x 2d, overlap d  Bin the features into the overlapping cells.  Each cell index  list of feature indices binned in this cell.  Each (x,y) point  4 cells.

d Grid based approach: For each of the K equidistant points,  Compute its distance from centers of 4 corresponding cells.  Select the closest cell.  Select all points in this cell as candidates for matching. Candidate Search: O(K) Geometry-aware Matching Efficiently finding the candidate set

Grid based approach: Candidate Search: O(K) Matching operations reduced: From (n log n) to (|c|log|c|) |c| ≈ ~ 0.8% of n for 2MP image. Geometry-aware Matching Efficiently finding the candidate set

l’ l F Geometry-aware Matching Efficiently finding the candidate set

 Fewer candidates, faster search (5-10x speed up).  Grid based approach is GPU-friendly.  Selective ratio-test:  Retains valid matches for repetitive elements  Denser and more complete point clouds Geometry-aware Matching

Datasets & Experiments  Matching Image Pairs of 4 different Scene type  Indoor, Plaza, Monument, Desk objects  Matches were verified by hand  Match-graph construction + SFM Reconstruction  Barcelona Museum, Tsinghua School, Notre Dame  ~ 90 – 200 images  Reconstruction by bundler

Key Results  CPU : ~ 5-10 times faster than Kd-tree matching  GPU : ~10 times faster than SIFTGPU matching.  Pairwise Matching : 2 – 5 times more matches  Reconstruction : 1.5 – 6 times more 3D points

Qualitative Results Unguided MatchingGeometry-aware Matching

Correctly Matched Points Unguided MatchingGeometry-aware Matching

Qualitative Results Unguided Matching Geometry-aware Matching 119/191 images 181/191 images

Thank You