The Correspondence Problem and “Interest Point” Detection Václav Hlaváč Center for Machine Perception Czech Technical University Prague Courtesy Martin Urban (the talk is based on his presentation) and Jana Kostkova, Jan Kybic, Jiri Matas, Radim Sara
2 1.The Correspondence Problem (CP) lies at the core of a number of computer vision problems: tracking (correspondence in consecutive frames narrow-baseline stereo wide-baseline stereo egomotion (camera motion) estimation motion segmentation in sequences with a moving camera recognition, categorisation, … 2.Demo of typical applications 3.The CP is commonly solved by robust matching of “Interest Points” 4.“Interest Points” are regions with distinguishing property, allowing there detection in a viewpoint and illumination invariant manner 5. Harris “Corner” (HC) Detection Algorithm HC are rotation and translation invariant interest points HC are interest points most commonly used in tracking and stereo Lecture Overview
3 “Ideal” solution: a pixel to pixel mapping from A to B (B ⋃ NULL) What it is “The correspondence problem”? A B Ex. result: x-shift
Applications: 3D Reconstruction
3D Reconstruction
Camera motion tracking ⇒ image stabilization original stabilized
Camera motion tracking ⇒ 3D animation
Motion keying / segmentation input sequence Background sequenceForeground sequence
Motion Detection Input: Output:
Medical imaging – image registration from the atlas test slice deform. field before registrationafter
11 “Ideal” solution: a pixel to pixel mapping from A to B The correspondence problem A B How to do it?
12 The correspondence problem A B Intuitive approach: For ∀ pxl ∈ A, find a pxl ∈ B with the most similar neighbourhood. Problems: - How to measure similarity of image patches? - undistinguishable regions (e.g. texture-less) - not surjective map (onto) due to occlusions - not bijective map (one to one) due to scale changes - huge data ⇒ very hard / impossible to recover the mapping by direct minimisation
13 The correspondence problem Conclusion: It is very hard / impossible to recover dense pxl 2 pxl mapping between two images. Solution: 1. Recover the correspondence relation just between several well distinguished image features (interest points / corners / Harris points). 2. Estimate multi-view transformation (e.g. epipolar geometry, camera motion). 3. Having epipolar geometry, try to find dense correspondences.
14 Corner Detection: Introduction Corner detector detects points with distinguished neighbourhood(*) well suited for matching verification.(*) undistinguished patches: distinguished patches:
15 - Demo of a point + with well distinguished neighbourhood. + > 0 Corner Detection: Introduction
16 - Demo of a point + with well distinguished neighbourhood. + > 0 Corner Detection: Introduction
17 - Demo of a point + with well distinguished neighbourhood. + > 0 Corner Detection: Introduction
18 - Demo of a point + with well distinguished neighbourhood. + > 0 Corner Detection: Introduction
19 - Demo of a point + with well distinguished neighbourhood. + > 0 Corner Detection: Introduction
20 - Demo of a point + with well distinguished neighbourhood. + > 0 Corner Detection: Introduction
21 - Demo of a point + with well distinguished neighbourhood. + > 0 Corner Detection: Introduction
22 - Demo of a point + with well distinguished neighbourhood. + > 0 Corner Detection: Introduction
23 Example of detected points Corner Detection: Introduction
24 Corner Detection: Basic principle undistinguished patches: distinguished patches: Image gradients ∇I(x,y) of undist. patches are (0,0) or have only one principle component. Image gradients ∇I(x,y) of dist. patches have two principle components. ⇒ rank ( ∑ ∇I(x,y)* ∇I(x,y) ⊤ ) = 2
25 Algorithm (C. Harris, 1988) 1. filter the image by gaussian (2x 1D convolution), sigma_d 2. compute the intensity gradients ∇I(x,y), (2x 1D conv.) 3. for each pixel and given neighbourhood, sigma_i: - compute auto-correlation matrix A = ∑ ∇I(x,y)* ∇I(x,y) ⊤ - and evaluate the response function R(A): R(A) >> 0 for rank(A)=2, R(A) → 0 for rank(A)<2 4. choose the best candidates (non-max suppression and thresholding)
26 Corner Detection: Algorithm (R. Harris, 1988) Harris response function R(A): R(A) = det (A) – k*trace 2 (A), [lamda1,lambda2] = eig(A)
27 Corner Detection: Algorithm (R. Harris, 1988) Algorithm properties: + “invariant” to 2D image shift and rotation + invariant to shift in illumination + “invariant” to small view point changes + low numerical complexity - not invariant to larger scale changes - not invariant to high contrast changes - not invariant to bigger view point changes
28 Corner Detection: Algorithm (C. Harris, 1988) Exp.: Harris points and view point change
29 Corner Detection: Harris points versus sigma_d and sigma_i Sigma_I → ↑ Sigma_d
30 Corner Detection: Application Algorithm: 1.Corner detection 2.Tentative correspondences - by comparing similarity of the corner neighb. in the searching window (e.g. cross-correlation) 3.Camera motion geometry estimation (e.g. by RANSAC) - finds the motion geometry and consistent correspondences 4. 3D reconstruction - triangulation, bundle adjustment 3D camera motion tracking / 3D reconstruction
31 Corner Detection: Camera Tracking Application - Boujou Input sequence
32 Corner Detection: Camera Tracking Application - Boujou Harris points
33 Corner Detection: Camera Tracking Application points consistent with 3D camera motion
34 Corner Detection: Camera Tracking Application 3D points and 3D camera motion
35 Corner Detection: Camera Tracking Application 3D animation
36 Corner Detection: Camera Tracking Application - Boujou Input sequence
37 Corner Detection: Camera Tracking Application - Boujou Harris points
38 Corner Detection: Camera Tracking Application - Boujou points consistent with 3D camera motion
39 for the presentation I copied almost done presntation Martin Urban for used demo images and software: Jana Kostková, CMP - (slide 3: images, disparity map) Radim Šára, CMP - (slide 5: images, 3D face reconstruction demo) Jan Kybic, CMP - (slide 10: medical image registration) 2d3 – (slide 31-38: Boujou Demo, slide 6: img. sequences) ImagineerSystems – (slide 8: MoKey Demo, slide 8: img. sequences) The Pixel Farm – (slide 31-33: img. sequence) Acknowledgements