1. Image Matching via Saliency Region Correspondences Alexander Toshev Jianbo Shi Kostas Daniilidis IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2007

2. Outline Introduction Joint-Image Graph (JIG) Matching Model Optimization in the JIG Estimation of Dense Correspondences Implementation Details Experiments Conclusion

3. Outline Introduction Joint-Image Graph (JIG) Matching Model Optimization in the JIG Estimation of Dense Correspondences Implementation Details Experiments Conclusion

4. Introduction Correspondence estimation is one of the fundamental challenges in computer vision lying in the core of many problems To find the correspondence of interest points, whose power is in the ability to robustly capture discriminative image structures

5. Introduction Feature-based approaches suffer from the ambiguity of local feature descriptors To address matching ambiguities is to provide grouping constraints via segmentation Disadvantage ： changing drastically even for small deformation of the scene

6. Introduction Example : Improvement : Matching by modeling in one score function both the coherence of regions

7. Introduction A pair of corresponding regions as co-salient define them as follows: Each region in the pair should exhibit strong internal coherence with respect to the background in the image The correspondence between the regions from the two images should be supported by high similarity of features extracted from these regions

8. Outline Introduction Joint-Image Graph (JIG) Matching Model Optimization in the JIG Estimation of Dense Correspondences Implementation Details Experiments Conclusion

9. Joint-Image Graph Matching Model To formalize this model Introduce the joint-image graph (JIG) which contains vertices the pixels of both images edges represent intra-image similarities and inter-image feature matches A good cluster in the JIG consists of a pair of coherent segments describing corresponding scene parts from the two image

10. Joint-Image Graph Matching Model

11. In order to combine the robustness of matching via local features with the descriptive power of salient segments We detect clusters in JIG represents a pair of co-salient regions contains pixels from both images : 1. coherent and perceptually salient regions in the images (intra-image similarity criterion) 2. match well according to the feature descriptors (inter-image similarity criterion)

12. Joint-Image Graph Matching Model Intra-image similarity : The image segmentation score is the Normalized Cut criterion applied to both segments (2)

13. Joint-Image Graph Matching Model Inter-image similarity : This function measures the strength of the connections between the regions and Correspondences between pixels are weakly connected with their neighboring pixels – exactly is uncertain If we use the same indicator vector, then it can be shown that (3)

14. Joint-Image Graph Matching Model The correspondence matrix is defined in terms of feature correspondences encoded in a matrix should select from pixel matches which connect each pixel of one of the images with at most one pixel of the other image This can be written as

15. Joint-Image Graph Matching Model Matching score function we should maximize the sum of the scores in eq. (2) and eq. (3) in the case of pairs of co-salient regions we can introduce indicator vectors packed in matrix we need to maximize subject to

16. Joint-Image Graph Matching Model The above optimization problem is NP-hard even for fixed We relax the indicator vectors to real numbers Following [12] it can be shown that the problem is equivalent to where is a matrix containing feature similarities across the images the constraints enforce to select for each pixel in one of the images only one pixel in the another which it can be mapped (4) [12] S. Yu and J. Shi. Multiclass spectral clustering. In ICCV,2003

17. Joint-Image Graph Matching Model

18. Outline Introduction Joint-Image Graph (JIG) Matching Model Optimization in the JIG Implementation Details Estimation of Dense Correspondences Experiments Conclusion

19. Optimization in the JIG In order to optimize matching score function we adopt an iterative two-step approach First step we maximize with respect to for given this step amounts to synchronization of the ’soft’ segmentations of two images based on Second step, we find an optimal correspondence matrix given the joint segmentation

20. Optimization in the JIG Segmentation synchronization for fixed the optimization problem from eq. (4) can be solved in a closed form – the maximum is attained for eigenvectors of the generalized eigenvalue problem due to clutter in this may lead to erroneous solutions assume that the joint ’soft’ segmentation lies in the subspace spanned by the ’soft’ segmentations and of the separate images where are eigenvectors of the corresponding generalized eigenvalue problems for each of the images

21. Optimization in the JIG Segmentation synchronization Hence we can write:,where is the joint image segmentation subspace basis and are the coordinates of the joint ’soft’ segmentation in this subspace With this subspace restriction for V the score function can be written as subject to is the original JIG weight matrix restricted to the segmentation subspaces (5)

22. Optimization in the JIG Segmentation synchronization If we write in terms of the subspace basis coordinates and for both image then the score function can be decomposed as follows: (6)

23. Optimization in the JIG Segmentation synchronization In eq. (6) The first term serves as a regularizer, which emphasizes eigenvectors in the subspaces with larger eigenvalues describing clearer segments The second term is a correlation between the segmentations of both images weighted by the correspondences in measures the quality of the match

24. Optimization in the JIG Segmentation synchronization The optimal in eq. (5) is attained for the eigenvectors of : diagonal matrix with the largest eigenvalues is a matrix, In eq. (4) has much higher dimension

25. Optimization in the JIG Segmentation synchronization

26. Optimization in the JIG Segmentation synchronization A different view of the above process can be obtained by representing the eigenvectors by their rows: denote by the row of We can assign to each pixel in the image a k-dimensional vector which we will call the embedding vector of this pixel The segmentation synchronization can be viewed as a rotation of the segmentation embeddings of both images such that corresponding pixels are close in the embedding

27. Optimization in the JIG Figure 4

28. Optimization in the JIG Obtaining discrete co-salient regions From the synchronized segmentation eigenvectors we can extract regions : suppose is the embedding vector of a particular pixel the binary mask which describes the segment is a column vector defined as describes a segment in the JIG and represents a pair of corresponding segments in the images the matching score between segments can be defined as

29. Optimization in the JIG Optimizing the correspondence matrix After we obtained we seek In order to obtain fast solution we relax the problem by removing the last inequality constrain we denote where is the embedded vector for pixel (eq. (4)) (7)

30. Optimization in the JIG Algorithm 1 1. Initialize. Compute 2. Compute segmentation subspaces as the eigenvectors to the largest eigenvalues of 3. Find optimal segmentation subspace alignment by computing the eigenvectors of 4. Compute optimal as in eq. (7). 5. If different from previous iteration go to step 3 6. Obtain pairs of corresponding segments is the match score for the co-salient region

31. Outline Introduction Joint-Image Graph (JIG) Matching Model Optimization in the JIG Estimation of Dense Correspondences Implementation Details Experiments Conclusion

32. Estimation of Dense Correspondences Initially we choose a sparse set of feature matches extracted using a feature detector In order to obtain denser set of correspondences we use a larger set of matches between features extracted everywhere in the image Since this set can potentially contain many more wrong matches than, running algorithm 1 directly on does not give always satisfactory results

33. Estimation of Dense Correspondences We prune based on the solution by combining Similarity between co-salient regions obtained for old feature set Using the embedding view of the segmentation synchronization from fig. 4 this translates to euclidean distances in the joint segmentation space weighted by the eigenvalues Feature similarity from new

34. Estimation of Dense Correspondences Suppose, two pixels and have embedding coordinates and obtained from Then following feature similarities embody both requirements from above: Finally, the entries in are scaled such that the largest value in is 1 The new co-salient regions are obtained as a solution of

35. Estimation of Dense Correspondences Algorithm 2 Matching algorithm 1. Extract conservatively using a feature detector 2. Solve using alg. 1 3. Extract using features extracted everywhere in the image 4. Compute and are the rows of Scale such that maximal element in is 1 5. Solve using alg. 1

36. Outline Introduction Joint-Image Graph (JIG) Matching Model Optimization in the JIG Estimation of Dense Correspondences Implementation Details Experiments Conclusion

37. Implementation Details Inter-image similarities The feature correspondence matrix is based on affine covariant region detector For comparison, each feature is represented by a descriptor extracted from be used to evaluate the appearance similarity between two interest points and

38. Implementation Details Inter-image similarities Define a similarity between pixels and lying in the interest point regions: 1 st term measures the appearance similarity between the regions in which and lie 2 nd term measures their geometric compatibility with respect to the affine transformation of to

39. Implementation Details Inter-image similarities Provided, we have extracted two feature sets from and from as described above the final match score for a pair of pixels equals the largest match score supported by a pair of feature points: pixels on different sides of corresponding image contours in both images get connected shape information is encoded in

40. Implementation Details Inter-image similarities

41. Implementation Details Inter-image similarities The final is obtained by pruning: retain For feature extraction we use the MSER detector [12] combined with SIFT descriptor [4] For the dense correspondences we use features extracted on a dense grid in the image and use the same descriptor [10] T. Tuytelaars and L. V. Gool. Matching widely separated views based on affine invariant regions. IJCV, 59(1):61–85,2004 [4] D. Lowe. Distinctive image features from scale-invariant keypoints. IJCV, 60(2), 91-110, 2004

42. Implementation Details Intra-image similarities The matrices for each image are based on intervening contours two pixels and from the same image belong to the same segment if there are no edges with large magnitude, which spatially separate them:

43. Implementation Details Algorithm settings The optimal dimension of the segmentation subspaces in step 2 depends on the area of the segments in the images -- to capture small detailed regions we need more eigenvectors For the experiments we used The threshold from is determined so that initially we obtain approx. 200 − 400 matches for our experiments it is

44. Implementation Details Time complexity denote by the time complexity of step 1,2 in alg. 1 corresponds to the complexity of the Ncut segmentation which is [12] the complexity of line 3 is computing the full SVD of a dense matrix of size denote the number of interest point matching is line 4 takes line 6 is

45. Implementation Details Time complexity in alg. 2, we use alg. 1 twice and step 4 is the total complexity of alg. 1 is we can precompute the segmentation for an image and use it every time we match this image

46. Outline Introduction Joint-Image Graph (JIG) Matching Model Optimization in the JIG Estimation of Dense Correspondences Implementation Details Experiments Conclusion

47. Experiments We conduct two experiments : 1. detection of matching regions 2. place recognition datasets : ICCV2005 Computer Vision Contest : Test4 & Final5 containing each 38 and 29 images of buildings each building is shown under different viewpoints

48. Experiments Detection of Matching Regions

49. Experiments Detection of Matching Regions

50. Experiments Detection of Matching Regions detect matching regions, enhance the feature matches, and segment common objects in manually selected image pairs the 30 matches with highest score in of the output the top 6 matching regions

51. Experiments Detection of Matching Regions Finding the correct match for a given point may fail usually because : 1. The appearance similarity to the matching point is not as high as the score of the best matches ( not ranked high in the initial ) 2. There are several matches with high scores due to similar or repeating structure

52. Experiments Detection of Matching Regions To compare quantitatively the difference between the initial and the improved set of feature matches we count how many of the top 30, 60, and 90 best matches are correct

53. Experiments Place Recognition Test4 and Final5 has been split into two subsets: exemplar set and query set The query set contains for Test4 19 and for Final5 22 images, while the exemplar set contains 9 and 16 images respectively Each query image is compared with all exemplars images and the matches are ranked according to the value of the match score function

54. Experiments Place Recognition For all queries, which have at least similar exemplars in the dataset compute how many of them are among the top matches

55. Outline Introduction Joint-Image Graph (JIG) Matching Model Optimization in the JIG Estimation of Dense Correspondences Implementation Details Experiments Conclusion

56. Present an algorithm to detects co-salient regions These regions are obtained through synchronization of the segmentations using local feature matches Dense correspondence between coherent segments are obtained The approach has shown promising results for correspondence detection in the context of place recognition