The Correspondence Problem and “Interest Point” Detection Václav Hlaváč Center for Machine Perception Czech Technical University Prague

Slides:



Advertisements
Similar presentations
Distinctive Image Features from Scale-Invariant Keypoints
Advertisements

Feature extraction: Corners
Feature Detection. Description Localization More Points Robust to occlusion Works with less texture More Repeatable Robust detection Precise localization.
3D Model Matching with Viewpoint-Invariant Patches(VIP) Reporter :鄒嘉恆 Date : 10/06/2009.
CSE 473/573 Computer Vision and Image Processing (CVIP)
Interest points CSE P 576 Ali Farhadi Many slides from Steve Seitz, Larry Zitnick.
TP14 - Local features: detection and description Computer Vision, FCUP, 2014 Miguel Coimbra Slides by Prof. Kristen Grauman.
MASKS © 2004 Invitation to 3D vision Lecture 7 Step-by-Step Model Buidling.
1 Interest Operators Find “interesting” pieces of the image –e.g. corners, salient regions –Focus attention of algorithms –Speed up computation Many possible.
(1) Feature-point matching by D.J.Duff for CompVis Online: Feature Point Matching Detection,
1 Interest Operator Lectures lecture topics –Interest points 1 (Linda) interest points, descriptors, Harris corners, correlation matching –Interest points.
Feature matching and tracking Class 5 Read Section 4.1 of course notes Read Shi and Tomasi’s paper on.
Midterm review: Cameras Pinhole cameras Vanishing points, horizon line Perspective projection equation, weak perspective Lenses Human eye Sample question:
Computing motion between images
Automatic Image Alignment (feature-based) : Computational Photography Alexei Efros, CMU, Fall 2005 with a lot of slides stolen from Steve Seitz and.
Feature tracking Class 5 Read Section 4.1 of course notes Read Shi and Tomasi’s paper on good features.
Distinctive image features from scale-invariant keypoints. David G. Lowe, Int. Journal of Computer Vision, 60, 2 (2004), pp Presented by: Shalomi.
Scale Invariant Feature Transform (SIFT)
CS664 Lecture #19: Layers, RANSAC, panoramas, epipolar geometry Some material taken from:  David Lowe, UBC  Jiri Matas, CMP Prague
Image Primitives and Correspondence
Automatic Image Alignment (feature-based) : Computational Photography Alexei Efros, CMU, Fall 2006 with a lot of slides stolen from Steve Seitz and.
1 Interest Operators Find “interesting” pieces of the image Multiple possible uses –image matching stereo pairs tracking in videos creating panoramas –object.
Summary of Previous Lecture A homography transforms one 3d plane to another 3d plane, under perspective projections. Those planes can be camera imaging.
Automatic Camera Calibration
Overview Introduction to local features
Computer vision: models, learning and inference
MASKS © 2004 Invitation to 3D vision Lecture 3 Image Primitives andCorrespondence.
Final Exam Review CS485/685 Computer Vision Prof. Bebis.
Lecture 12 Stereo Reconstruction II Lecture 12 Stereo Reconstruction II Mata kuliah: T Computer Vision Tahun: 2010.
1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.
Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia.
Course Syllabus 1.Color 2.Camera models, camera calibration 3.Advanced image pre-processing Line detection Corner detection Maximally stable extremal regions.
Overview Harris interest points Comparing interest points (SSD, ZNCC, SIFT) Scale & affine invariant interest points Evaluation and comparison of different.
Local invariant features Cordelia Schmid INRIA, Grenoble.
Local Invariant Features This is a compilation of slides by: Darya Frolova, Denis Simakov,The Weizmann Institute of Science Jiri Matas, Martin Urban Center.
Imaging Geometry for the Pinhole Camera Outline: Motivation |The pinhole camera.
CSCE 643 Computer Vision: Structure from Motion
3D Reconstruction Jeff Boody. Goals ● Reconstruct 3D models from a sequence of at least two images ● No prior knowledge of the camera or scene ● Use the.
Computer Vision 776 Jan-Michael Frahm 12/05/2011 Many slides from Derek Hoiem, James Hays.
Lec 22: Stereo CS4670 / 5670: Computer Vision Kavita Bala.
776 Computer Vision Jan-Michael Frahm, Enrique Dunn Spring 2013.
18 th August 2006 International Conference on Pattern Recognition 2006 Epipolar Geometry from Two Correspondences Michal Perďoch, Jiří Matas, Ondřej Chum.
Local invariant features Cordelia Schmid INRIA, Grenoble.
Harris Corner Detector & Scale Invariant Feature Transform (SIFT)
Edges.
Jack Pinches INFO410 & INFO350 S INFORMATION SCIENCE Computer Vision I.
Features, Feature descriptors, Matching Jana Kosecka George Mason University.
Project 3 questions? Interest Points and Instance Recognition Computer Vision CS 143, Brown James Hays 10/21/11 Many slides from Kristen Grauman and.
55:148 Digital Image Processing Chapter 11 3D Vision, Geometry Topics: Basics of projective geometry Points and hyperplanes in projective space Homography.
Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.
Local features: detection and description
Features Jan-Michael Frahm.
Lecture 9 Feature Extraction and Motion Estimation Slides by: Michael Black Clark F. Olson Jean Ponce.
MASKS © 2004 Invitation to 3D vision Lecture 3 Image Primitives andCorrespondence.
Correspondence and Stereopsis. Introduction Disparity – Informally: difference between two pictures – Allows us to gain a strong sense of depth Stereopsis.
Invariant Local Features Image content is transformed into local feature coordinates that are invariant to translation, rotation, scale, and other imaging.
SIFT.
SIFT Scale-Invariant Feature Transform David Lowe
Lecture 07 13/12/2011 Shai Avidan הבהרה: החומר המחייב הוא החומר הנלמד בכיתה ולא זה המופיע / לא מופיע במצגת.
3D Vision Interest Points.
TP12 - Local features: detection and description
Local features: detection and description May 11th, 2017
Image Primitives and Correspondence
Local Invariant Features
What have we learned so far?
CSE 455 – Guest Lectures 3 lectures Contact Interest points 1
Presented by Xu Miao April 20, 2005
Recognition and Matching based on local invariant features
Presentation transcript:

The Correspondence Problem and “Interest Point” Detection Václav Hlaváč Center for Machine Perception Czech Technical University Prague Courtesy Martin Urban (the talk is based on his presentation) and Jana Kostkova, Jan Kybic, Jiri Matas, Radim Sara

2 1.The Correspondence Problem (CP) lies at the core of a number of computer vision problems: tracking (correspondence in consecutive frames narrow-baseline stereo wide-baseline stereo egomotion (camera motion) estimation motion segmentation in sequences with a moving camera recognition, categorisation, … 2.Demo of typical applications 3.The CP is commonly solved by robust matching of “Interest Points” 4.“Interest Points” are regions with distinguishing property, allowing there detection in a viewpoint and illumination invariant manner 5. Harris “Corner” (HC) Detection Algorithm HC are rotation and translation invariant interest points HC are interest points most commonly used in tracking and stereo Lecture Overview

3 “Ideal” solution: a pixel to pixel mapping from A to B (B ⋃ NULL) What it is “The correspondence problem”? A B Ex. result: x-shift

Applications: 3D Reconstruction

3D Reconstruction

Camera motion tracking ⇒ image stabilization original stabilized

Camera motion tracking ⇒ 3D animation

Motion keying / segmentation input sequence Background sequenceForeground sequence

Motion Detection Input: Output:

Medical imaging – image registration from the atlas test slice deform. field before registrationafter

11 “Ideal” solution: a pixel to pixel mapping from A to B The correspondence problem A B How to do it?

12 The correspondence problem A B Intuitive approach: For ∀ pxl ∈ A, find a pxl ∈ B with the most similar neighbourhood. Problems: - How to measure similarity of image patches? - undistinguishable regions (e.g. texture-less) - not surjective map (onto) due to occlusions - not bijective map (one to one) due to scale changes - huge data ⇒ very hard / impossible to recover the mapping by direct minimisation

13 The correspondence problem Conclusion: It is very hard / impossible to recover dense pxl 2 pxl mapping between two images. Solution: 1. Recover the correspondence relation just between several well distinguished image features (interest points / corners / Harris points). 2. Estimate multi-view transformation (e.g. epipolar geometry, camera motion). 3. Having epipolar geometry, try to find dense correspondences.

14 Corner Detection: Introduction Corner detector detects points with distinguished neighbourhood(*) well suited for matching verification.(*) undistinguished patches: distinguished patches:

15 - Demo of a point + with well distinguished neighbourhood. + > 0 Corner Detection: Introduction

16 - Demo of a point + with well distinguished neighbourhood. + > 0 Corner Detection: Introduction

17 - Demo of a point + with well distinguished neighbourhood. + > 0 Corner Detection: Introduction

18 - Demo of a point + with well distinguished neighbourhood. + > 0 Corner Detection: Introduction

19 - Demo of a point + with well distinguished neighbourhood. + > 0 Corner Detection: Introduction

20 - Demo of a point + with well distinguished neighbourhood. + > 0 Corner Detection: Introduction

21 - Demo of a point + with well distinguished neighbourhood. + > 0 Corner Detection: Introduction

22 - Demo of a point + with well distinguished neighbourhood. + > 0 Corner Detection: Introduction

23 Example of detected points Corner Detection: Introduction

24 Corner Detection: Basic principle undistinguished patches: distinguished patches: Image gradients ∇I(x,y) of undist. patches are (0,0) or have only one principle component. Image gradients ∇I(x,y) of dist. patches have two principle components. ⇒ rank ( ∑ ∇I(x,y)* ∇I(x,y) ⊤ ) = 2

25 Algorithm (C. Harris, 1988) 1. filter the image by gaussian (2x 1D convolution), sigma_d 2. compute the intensity gradients ∇I(x,y), (2x 1D conv.) 3. for each pixel and given neighbourhood, sigma_i: - compute auto-correlation matrix A = ∑ ∇I(x,y)* ∇I(x,y) ⊤ - and evaluate the response function R(A): R(A) >> 0 for rank(A)=2, R(A) → 0 for rank(A)<2 4. choose the best candidates (non-max suppression and thresholding)

26 Corner Detection: Algorithm (R. Harris, 1988) Harris response function R(A): R(A) = det (A) – k*trace 2 (A), [lamda1,lambda2] = eig(A)

27 Corner Detection: Algorithm (R. Harris, 1988) Algorithm properties: + “invariant” to 2D image shift and rotation + invariant to shift in illumination + “invariant” to small view point changes + low numerical complexity - not invariant to larger scale changes - not invariant to high contrast changes - not invariant to bigger view point changes

28 Corner Detection: Algorithm (C. Harris, 1988) Exp.: Harris points and view point change

29 Corner Detection: Harris points versus sigma_d and sigma_i Sigma_I → ↑ Sigma_d

30 Corner Detection: Application Algorithm: 1.Corner detection 2.Tentative correspondences - by comparing similarity of the corner neighb. in the searching window (e.g. cross-correlation) 3.Camera motion geometry estimation (e.g. by RANSAC) - finds the motion geometry and consistent correspondences 4. 3D reconstruction - triangulation, bundle adjustment 3D camera motion tracking / 3D reconstruction

31 Corner Detection: Camera Tracking Application - Boujou Input sequence

32 Corner Detection: Camera Tracking Application - Boujou Harris points

33 Corner Detection: Camera Tracking Application points consistent with 3D camera motion

34 Corner Detection: Camera Tracking Application 3D points and 3D camera motion

35 Corner Detection: Camera Tracking Application 3D animation

36 Corner Detection: Camera Tracking Application - Boujou Input sequence

37 Corner Detection: Camera Tracking Application - Boujou Harris points

38 Corner Detection: Camera Tracking Application - Boujou points consistent with 3D camera motion

39 for the presentation I copied almost done presntation Martin Urban for used demo images and software: Jana Kostková, CMP - (slide 3: images, disparity map) Radim Šára, CMP - (slide 5: images, 3D face reconstruction demo) Jan Kybic, CMP - (slide 10: medical image registration) 2d3 – (slide 31-38: Boujou Demo, slide 6: img. sequences) ImagineerSystems – (slide 8: MoKey Demo, slide 8: img. sequences) The Pixel Farm – (slide 31-33: img. sequence) Acknowledgements