Lecture 21: Multiple-view geometry and structure from motion

Slides:



Advertisements
Similar presentations
Image Rectification for Stereo Vision
Advertisements

The fundamental matrix F
Lecture 11: Two-view geometry
Jan-Michael Frahm, Enrique Dunn Spring 2012
Two-view geometry.
Lecture 8: Stereo.
Dr. Hassan Foroosh Dept. of Computer Science UCF
Camera calibration and epipolar geometry
Stanford CS223B Computer Vision, Winter 2005 Lecture 5: Stereo I Sebastian Thrun, Stanford Rick Szeliski, Microsoft Hendrik Dahlkamp and Dan Morris, Stanford.
Lecture 12: Structure from motion CS6670: Computer Vision Noah Snavely.
Lecture 11: Structure from motion, part 2 CS6670: Computer Vision Noah Snavely.
Stanford CS223B Computer Vision, Winter 2006 Lecture 5 Stereo I
Epipolar geometry. (i)Correspondence geometry: Given an image point x in the first view, how does this constrain the position of the corresponding point.
Uncalibrated Geometry & Stratification Sastry and Yang
Multiple View Geometry
Lecture 22: Structure from motion CS6670: Computer Vision Noah Snavely.
Lecture 20: Two-view geometry CS6670: Computer Vision Noah Snavely.
3D Computer Vision and Video Computing 3D Vision Lecture 14 Stereo Vision (I) CSC 59866CD Fall 2004 Zhigang Zhu, NAC 8/203A
Projected image of a cube. Classical Calibration.
May 2004Stereo1 Introduction to Computer Vision CS / ECE 181B Tuesday, May 11, 2004  Multiple view geometry and stereo  Handout #6 available (check with.
Lec 21: Fundamental Matrix
CSE473/573 – Stereo Correspondence
Lecture 10: Robust fitting CS4670: Computer Vision Noah Snavely.
Multiple View Geometry
Multiple View Geometry. THE GEOMETRY OF MULTIPLE VIEWS Reading: Chapter 10. Epipolar Geometry The Essential Matrix The Fundamental Matrix The Trifocal.
CS 558 C OMPUTER V ISION Lecture IX: Dimensionality Reduction.
3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective.
Epipolar Geometry and Stereo Vision Computer Vision CS 543 / ECE 549 University of Illinois Derek Hoiem 04/12/11 Many slides adapted from Lana Lazebnik,
55:148 Digital Image Processing Chapter 11 3D Vision, Geometry Topics: Basics of projective geometry Points and hyperplanes in projective space Homography.
CSE 6367 Computer Vision Stereo Reconstruction Camera Coordinate Transformations “Everything should be made as simple as possible, but not simpler.” Albert.
Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates.
Stereo Readings Trucco & Verri, Chapter 7 –Read through 7.1, 7.2.1, 7.2.2, 7.3.1, 7.3.2, and 7.4, –The rest is optional. Single image stereogram,
Automatic Camera Calibration
Epipolar Geometry and Stereo Vision Computer Vision CS 543 / ECE 549 University of Illinois Derek Hoiem 03/05/15 Many slides adapted from Lana Lazebnik,
Computer vision: models, learning and inference
Epipolar Geometry and Stereo Vision Computer Vision ECE 5554 Virginia Tech Devi Parikh 10/15/13 These slides have been borrowed from Derek Hoiem and Kristen.
Multi-view geometry.
1 Preview At least two views are required to access the depth of a scene point and in turn to reconstruct scene structure Multiple views can be obtained.
Projective cameras Motivation Elements of Projective Geometry Projective structure from motion Planches : –
Recap from Monday Image Warping – Coordinate transforms – Linear transforms expressed in matrix form – Inverse transforms useful when synthesizing images.
Epipolar geometry Epipolar Plane Baseline Epipoles Epipolar Lines
Lecture 04 22/11/2011 Shai Avidan הבהרה : החומר המחייב הוא החומר הנלמד בכיתה ולא זה המופיע / לא מופיע במצגת.
Multiview Geometry and Stereopsis. Inputs: two images of a scene (taken from 2 viewpoints). Output: Depth map. Inputs: multiple images of a scene. Output:
Stereo Course web page: vision.cis.udel.edu/~cv April 11, 2003  Lecture 21.
Lec 22: Stereo CS4670 / 5670: Computer Vision Kavita Bala.
Announcements Project 3 due Thursday by 11:59pm Demos on Friday; signup on CMS Prelim to be distributed in class Friday, due Wednesday by the beginning.
Last lecture Passive Stereo Spacetime Stereo Multiple View Stereo.
EECS 274 Computer Vision Stereopsis.
Geometry of Multiple Views
Two-view geometry. Epipolar Plane – plane containing baseline (1D family) Epipoles = intersections of baseline with image planes = projections of the.
stereo Outline : Remind class of 3d geometry Introduction
Feature Matching. Feature Space Outlier Rejection.
Solving for Stereo Correspondence Many slides drawn from Lana Lazebnik, UIUC.
Rectification of Stereo Images Милош Миловановић Урош Поповић.
Geometry Reconstruction March 22, Fundamental Matrix An important problem: Determine the epipolar geometry. That is, the correspondence between.
Stereo March 8, 2007 Suggested Reading: Horn Chapter 13.
Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches : –
Lec 26: Fundamental Matrix CS4670 / 5670: Computer Vision Kavita Bala.
CSE 185 Introduction to Computer Vision Stereo 2.
Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates.
55:148 Digital Image Processing Chapter 11 3D Vision, Geometry
Epipolar Geometry and Stereo Vision
CS4670 / 5670: Computer Vision Kavita Bala Lec 27: Stereo.
Two-view geometry Computer Vision Spring 2018, Lecture 10
Epipolar geometry.
Epipolar geometry continued
Reconstruction.
Two-view geometry.
Two-view geometry.
Multi-view geometry.
Presentation transcript:

Lecture 21: Multiple-view geometry and structure from motion CS6670: Computer Vision Noah Snavely Lecture 21: Multiple-view geometry and structure from motion

Readings Szeliski, Chapter 7.1 – 7.4

Announcements Project 2b due next Tuesday, Nov 2, by 10:59pm Final project proposals due by this Friday, Oct 29, by 11:59pm See project webpage for project ideas

3d point lies somewhere along r Two-view geometry Where do epipolar lines come from? 3d point lies somewhere along r epipolar plane epipolar line epipolar line (projection of r) Image 1 Image 2

Fundamental matrix epipolar plane epipolar line epipolar line (projection of ray) Image 1 Image 2 This epipolar geometry of two views is described by a Very Special 3x3 matrix , called the fundamental matrix maps (homogeneous) points in image 1 to lines in image 2! The epipolar line (in image 2) of point p is: Epipolar constraint on corresponding points:

Fundamental matrix epipolar plane epipolar line epipolar line (projection of ray) Two special points: e1 and e2 (the epipoles): projection of one camera into the other

Relationship with homography? Images taken from the same center of projection? Use a homography, and map points directly to points! Images taken from different places? We need an F-matrix.

Fundamental matrix – uncalibrated case : intrinsics of camera 1 : intrinsics of camera 2 : rotation of image 2 w.r.t. camera 1 the Fundamental matrix

Cross-product as linear operator Useful fact: Cross product with a vector t can be represented as multiplication with a (skew-symmetric) 3x3 matrix

Fundamental matrix – calibrated case : ray through p in camera 1’s (and world) coordinate system : ray through q in camera 2’s coordinate system { the Essential matrix

Properties of the Fundamental Matrix is the epipolar line associated with and is rank 2 How many parameters does F have? T

Rectified case

Stereo image rectification reproject image planes onto a common plane parallel to the line between optical centers pixel motion is horizontal after this transformation two homographies (3x3 transform), one for each input image reprojection C. Loop and Z. Zhang. Computing Rectifying Homographies for Stereo Vision. IEEE Conf. Computer Vision and Pattern Recognition, 1999.

Questions?

Estimating F If we don’t know K1, K2, R, or t, can we estimate F for two images? Yes, given enough correspondences

Estimating F – 8-point algorithm The fundamental matrix F is defined by for any pair of matches x and x’ in two images. Let x=(u,v,1)T and x’=(u’,v’,1)T, each match gives a linear equation

8-point algorithm In reality, instead of solving , we seek f to minimize , least eigenvector of .

8-point algorithm – Problem? F should have rank 2 – not a linear constraint To enforce that F is of rank 2, F is replaced by F’ that minimizes subject to the rank constraint. This is achieved by SVD. Let , where , let then is the solution.

8-point algorithm % Build the constraint matrix A = [x2(1,:)‘.*x1(1,:)' x2(1,:)'.*x1(2,:)' x2(1,:)' ... x2(2,:)'.*x1(1,:)' x2(2,:)'.*x1(2,:)' x2(2,:)' ... x1(1,:)' x1(2,:)' ones(npts,1) ]; [U,D,V] = svd(A); % Extract fundamental matrix from the column of V % corresponding to the smallest singular value. F = reshape(V(:,9),3,3)'; % Enforce rank2 constraint [U,D,V] = svd(F); F = U*diag([D(1,1) D(2,2) 0])*V';

8-point algorithm Pros: it is linear, easy to implement and fast Cons: susceptible to noise In practice, we use the normalized 8-point algorithm (See R. Hartley, “In Defense of the Eight-Point Algorithm”, PAMI ’97).

Results (ground truth)

Results (8-point algorithm)

Results (normalized 8-point algorithm)

Estimating the F-matrix If we have more than eight points, we can solve a least-squares version of this problem What could go wrong? How can we fix this?

Estimating the F-matrix Extra bonus: If we run RANSAC to find an F-matrix with the most inliers, we can throw out inconsistent matches (i.e., this is a “bad match filter”) Might be useful later on…

What about more than two views? The geometry of three views is described by a 3 x 3 x 3 tensor called the trifocal tensor The geometry of four views is described by a 3 x 3 x 3 x 3 tensor called the quadrifocal tensor After this it starts to get complicated…