1. Geometry-aware Feature Matching for Structure from Motion Applications Rajvi Shah, Vanshika Srivastava, P J Narayanan Center for Visual Information Technology IIIT Hyderabad, India

2. Feature Matching for SfM Applications Match features across all image pairs

3. Pair-wise Feature Matching Image 1 + SIFTImage 2 + SIFT Img1 Descriptors Img2 Descriptors

4. O(M 2 ) Exhaustive Comparison Pair-wise Feature Matching Image 1 + SIFTImage 2 + SIFT Img1 Descriptors Img2 Descriptors

5. O(M log M) Kd-tree based Comparison Pair-wise Feature Matching Image 1 + SIFTImage 2 + SIFT Img1 Descriptors Img2 Descriptors

6. Features in Image 1 Features in Image 2Kd-tree of features in Image 2 (in 128-d descriptor space) Q Best Match2 nd Best Match Kd-tree based Feature Matching Search for top-two near neighbors Ratio of distances between query and 2-NNs

7. Features in Image 1 Features in Image 2Kd-tree of features in Image 2 (in 128-d descriptor space) Q Kd-tree based Feature Matching Best Match2 nd Best Match

8. Features in Image 1 Features in Image 2 Q Match F-matrix l = F.Q l Geometric Verification Distance of the matching point from epipolar line

9. Features in Image 1 Features in Image 2 Q F-matrix l = F.Q Distance from l > d Match l Incrorect Match d Geometric Verification Distance of the matching point from epipolar line

10. Main Problems  Kd-tree based approach: O(M log M)  Images with features in order on 10 4 ~ 10 5.  Wasted effort in matching inconsistent features  Difficult to parallelize  Ratio test is punitive for repetitive structures  Features have similar appearance, e.g. windows.  Salient but not distinctive to pass the ratio test  Rejection of many valid correspondences

11. Geometry-aware Matching Features in Image 1 Features in Image 2 1. Select a small subset (10%-20%) of features

12. Geometry-aware Matching Features in Image 1 Features in Image 2 2. Match the selected features using Kd-tree

13. Geometry-aware Matching Features in Image 1 Features in Image 2 3. Estimate F-Matrix using initial matches (Utilize it for geometry-guided matching) F-matrix

14. Features in Image 1 Features in Image 2 Q F-matrix l = F.Q Points with Distance from l < d Geometry-aware Matching 4. Find a set of candidate matches l

15. Features in Image 1 Features in Image 2 Q F-matrix l = F.Q Points with Distance from l < d Geometry-aware Matching 5. Find the top 2-NN within candidates & ratio test l

16. Features in Image 1 Features in Image 2 Q F-matrix l = F.Q Points with Distance from l < d Geometry-aware Matching Candidate set is different for each query! l

17. Geometry-aware Matching Efficiently finding the candidate set Distances of n points from the line l need to be computed. Linear Search: O(n)

18. d Geometry-aware Matching Efficiently finding the candidate set Distances of n points from the line l need to be computed. Linear Search: O(n) Select K points on line l, each at distance d. Radial Search: O(K log n) K << N.

19. d Geometry-aware Matching Efficiently finding the candidate set Distances of n points from the line l need to be computed. Linear Search: O(n) Select K points on line l, each at distance d. Radial Search: O(K log n) K << N.

20. d Geometry-aware Matching Efficiently finding the candidate set Distances of n points from the line l need to be computed. Linear Search: O(n) Select K points on line l, each at distance d. Radial Search: O(K log n) K << N. Optimize Further

21. d Grid based approach:  Divide the image using 4 grids of cell size 2d x 2d, overlap d G1 Geometry-aware Matching Efficiently finding the candidate set

22. d Grid based approach:  Divide the image using 4 grids of cell size 2d x 2d, overlap d G2 Geometry-aware Matching Efficiently finding the candidate set

23. d Grid based approach:  Divide the image using 4 grids of cell size 2d x 2d, overlap d G3 Geometry-aware Matching Efficiently finding the candidate set

24. d G4 Geometry-aware Matching Efficiently finding the candidate set Grid based approach:  Divide the image using 4 grids of cell size 2d x 2d, overlap d  Bin the features into the overlapping cells.  Each cell index  list of feature indices binned in this cell.  Each (x,y) point  4 cells.

25. d Grid based approach: For each of the K equidistant points,  Compute its distance from centers of 4 corresponding cells.  Select the closest cell.  Select all points in this cell as candidates for matching. Candidate Search: O(K) Geometry-aware Matching Efficiently finding the candidate set

26. Grid based approach: Candidate Search: O(K) Matching operations reduced: From (n log n) to (|c|log|c|) |c| ≈ ~ 0.8% of n for 2MP image. Geometry-aware Matching Efficiently finding the candidate set

27. l’ l F Geometry-aware Matching Efficiently finding the candidate set

28.  Fewer candidates, faster search (5-10x speed up).  Grid based approach is GPU-friendly.  Selective ratio-test:  Retains valid matches for repetitive elements  Denser and more complete point clouds Geometry-aware Matching

29. Datasets & Experiments  Matching Image Pairs of 4 different Scene type  Indoor, Plaza, Monument, Desk objects  Matches were verified by hand  Match-graph construction + SFM Reconstruction  Barcelona Museum, Tsinghua School, Notre Dame  ~ 90 – 200 images  Reconstruction by bundler

30. Key Results  CPU : ~ 5-10 times faster than Kd-tree matching  GPU : ~10 times faster than SIFTGPU matching.  Pairwise Matching : 2 – 5 times more matches  Reconstruction : 1.5 – 6 times more 3D points

31. Qualitative Results Unguided MatchingGeometry-aware Matching

32. Correctly Matched Points Unguided MatchingGeometry-aware Matching

33. Qualitative Results Unguided Matching Geometry-aware Matching 119/191 images 181/191 images

34. Thank You rajvi.shah@research.iiit.ac.in http://cvit.iiit.ac.in/multistagesfm/