Special Topic on Image Retrieval Local Feature Matching Verification

Geometric Verification Motivation –Remove false matches by checking geometric consistency 2 Red line: geometric consistent match Blue line: geometric inconsistent match

Global Verification: RANSAC Take RANSAC as an example – Check geometric consistency from matched feature pairs. Random sampling

Local Geometric-Verification Locally nearest neighbors （ Video Goole, cvpr’03 ） – Matched regions should have a similar spatial layout. – For each match define its search area – Region in the search area that also matches casts a vote for the image – Reject matches with no support Drawback – Sensitive to clutter

Hamming Embedding (ECCV’08) Introduced as an extension of BOV [Jegou 08] – Combination of – A partitioning technique (k-means) – A binary code that refine the descriptor Representation of a descriptor x – Vector-quantized to q(x) as in standard BOV – short binary vector b(x) for an additional localization in the Voronoi cell Two descriptors x and y match iif

Hamming Embedding Binary signature generation – Off-line learning Random matrix generation Descriptor projection and assignment Median values of projected descriptors – On-line binarization Quantization assignment Descriptor projection Computing the signature:

Local Geometric-Verification Bundled feature (CVPR’09) – Group local features in local MSER region. – Increase discriminative power of visual words. – Allowed to have large overlap error. Bundle comparison: – M m (q; p): number of common visual words between two bundles – M g (q; p): inconsistency of geometric order in x- and y- direction. Drawbacks: Infeasible for rotated bundles.

– Visual words are bundled in MSER regions. – Spatial consistency for bundled features is utilized to weight visual words. Z. Wu, J. Sun, and Q. Ke, “Bundling Features for Large Scale Partial-Duplicate Web Image Search,” CVPR 09 # of shared visual words Spatial consistency – Great performance for partial-dup detection in over 1 M database – Drawbacks: Infeasible for rotated bundles. Local Geometric-Verification Bundled feature (CVPR’09)

Global Verification: RANSAC  RANSAC: remove outliers by inlier classification  Inliers: true matched features  Outliers: false matched features  Assumption of RANdom SAmple Consensus (RANSAC)  The original data consists of inliers and outliers.  A subset of inliers can estimate a model to optimally explain the inliers.  Estimate the affine transformation by RANSAC  Procedure: Iteratively select a random subset as hypothetical inliers 1.A model is fitted to the hypothetical inliers. 2.All other data are tested against the fitted model for inlier classification. 3.The model is re-estimated from all hypothetical inliers. 4.The model is evaluated by estimating the error of the inliers relative to the model.  Drawbacks: Computationally expensive, not scalable Fischler, et al., RANdom SAmple Consensus: a paradigm for model fitting with applications to image analysis and automated cartography, Comm. of the ACM, 24: , 1981

Spatial Coding for Geometric Verification (ACM MM’ 10) Motivation – Encode local features’ relative positions into compact binary maps – Check spatial consistency of local matches for geometric verification Spatial coding maps – Relative spatial positions between local features. – Very efficient and high precision Zhou & Tian, Spatial Coding for large scale partial-duplicate image search. ACM Multimedia 2010.

Spatial Map Generation Rotate 45 degree counterclockwise  In previous case, each quadrant has one part  Consider each quadrant is uniformly divided into two parts. 11 =

Spatial Map Generation  Generalized spatial map: GX and GY  Each quadrant is uniformly divided into r parts. 12 … … k=r-1 k=1 k=0 X-map Y-map

Generalized Spatial Coding  Spatial coding maps:  Each quadrant uniformly divided into r parts.  Decompose the division into r sub-division.  Rotate each sub-division to align the axis. New feature locations after rotation ： Generalized spatial maps ： 13

Spatial Verification  Verification with spatial maps GX and GY  Compare the spatial maps of matched features:  k=0, …, r-1; i, j=1, …, N; N: number of matched features  Find and delete the most inconsistent matched pair, recursively: 14 V x : inconsistent degree in X-map V y : inconsistent degree in Y-map Identify i* and remove

x y

Geometric Verification with Coding Maps 16 SUM x y

Image Plane Division (TOMCCAP’ 10) (a) (d) (c) (b) (e) (f)

Geometric Square Coding Coordinate adjustment Square coding map Generalized map ：

Geometric Fan Coding Fan coding maps Coordinate adjustment Generalized coding maps 19

Geometric Verification Compare the fan coding maps of matched features: Inconsistency measurement from geometric fan coding: Inconsistency measurement from geometric square coding: Inconsistency matrix: