Stereo and Epipolar geometry Jana Kosecka CS223b

Previously Image Primitives (feature points, lines, contours) Today: How to match primitives between two (multiple) views) Goals: 3D reconstruction, recognition Stereo matching and reconstruction (canonical configuration) Epipolar Geometry (general two view setting) CS223b

2D images project 3D points into 2D: Why Stereo Vision? 2D images project 3D points into 2D: P Q P’=Q’ O center of projection 3D points on the same viewing line have the same 2D image: 2D imaging results in depth information loss CS223b

Canonical Stereo Configuration Assumes (two) cameras Known positions and focal lengths Recover depth P f O1 O2 T CS223b

Depth can be recovered by triangulation Prerequisite: Finding Correspondences Given an image point in left image, the (corresponding) point in the right image, is the point which is the projection of the same 3-D point CS223b

Random dot stereograms B. Julesz: Showed that correspondence is not needed for stereo. CS223b

Autostereograms Depth perception from one image Viewing trick the brain by focusing at the plane behind - match can be established perception of 3D CS223b

Correspondence Problem Two classes of algorithms: Correlation-based algorithms Produce a DENSE set of correspondences Feature-based algorithms Produce a SPARSE set of correspondences CS223b

Stereo – Photometric Constraint Same world point has same intensity in both images. Lambertian fronto-parallel Issues (noise, specularities, foreshortening) Difficulties – ambiguities, large changes of appearance, due to change Of viewpoint, non-uniquess CS223b

Stereo Matching What if ? For each scanline , for each pixel in the left image compare with every pixel on same epipolar line in right image pick pixel with minimum match cost This will never work, so: improvement match windows CS223b (slides O. Camps)

Comparing Windows: ? = g f Most popular For each window, match to closest window on epipolar line in other image. (slides O. Camps) CS223b

Comparing Windows: Minimize Sum of Squared Differences Maximize Cross correlation It is closely related to the SSD: CS223b

Region based Similarity Metrics Sum of squared differences Normalize cross-correlation Sum of absolute differences CS223b

NCC score for two widely separated views CS223b

Window size W = 3 W = 20 Better results with adaptive window Effect of window size smaller window: more detail, more noise bigger window: less noise, more detail Better results with adaptive window T. Kanade and M. Okutomi, A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment,, Proc. International Conference on Robotics and Automation, 1991. D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffusion. International Journal of Computer Vision, 28(2):155-174, July 1998 (S. Seitz) CS223b

Data from University of Tsukuba Stereo results Data from University of Tsukuba Scene Ground truth (Seitz) CS223b

Results with window correlation Window-based matching (best window size) Ground truth (Seitz) CS223b

Results with better method State of the art Ground truth Boykov et al., Fast Approximate Energy Minimization via Graph Cuts, International Conference on Computer Vision, September 1999. (Seitz) CS223b

More of advanced stereo Ordering constraint Dynamic programming Global optimization – next lecture CS223b

Two view – General Configuration Motion between the two views is not known Given two views of the scene recover the unknown camera displacement and 3D scene structure CS223b

Pinhole Camera Imaging Model 3D points Image points Perspective Projection Rigid Body Motion Rigid Body Motion + Persp. projection CS223b

Rigid Body Motion – Two Views CS223b

3D Structure and Motion Recovery Euclidean transformation Measurements Corresponding points unknowns Find such Rotation and Translation and Depth that the reprojection error is minimized Two views ~ 200 points 6 unknowns – Motion 3 Rotation, 3 Translation - Structure 200x3 coordinates - (-) universal scale Difficult optimization problem CS223b

Notation Cross product between two vectors in where CS223b

Epipolar Geometry Algebraic Elimination of Depth Essential matrix Image correspondences Algebraic Elimination of Depth [Longuet-Higgins ’81]: Essential matrix CS223b

Additional constraints Epipolar Geometry Epipolar lines Epipoles Image correspondences Epipolar transfer Additional constraints CS223b

Characterization of Essential Matrix special 3x3 matrix (Essential Matrix Characterization) A non-zero matrix is an essential matrix iff its SVD: satisfies: with and and CS223b

Estimating Essential Matrix Find such Rotation and Translation that the epipolar error is minimized Space of all Essential Matrices is 5 dimensional 3 DOF Rotation, 2 DOF – Translation (up to scale !) Denote Rewrite Collect constraints from all points CS223b

Estimating Essential Matrix Solution Eigenvector associated with the smallest eigenvalue of If degenerate configuration estimated using linear least squares unstack -> Projection on to Essential Space (Project onto a space of Essential Matrices) If the SVD of a matrix is given by then the essential matrix which minimizes the Frobenius distance is given by with CS223b

Pose Recovery from Essential Matrix There are two relative poses with and corresponding to a non-zero matrix essential matrix. Twisted pair ambiguity CS223b

Two view linear algorithm - summary Solve the LLSE problem: Solution eigenvector associated with smallest eigenvalue of Compute SVD of F recovered from data E is 5 diml. sub. mnfld. in 8-point linear algorithm Project onto the essential manifold: Recover the unknown pose: CS223b

Pose Recovery There are two pairs corresponding to essential matrix . Positive depth constraint disambiguates the impossible solutions Translation has to be non-zero Points have to be in general position - degenerate configurations – planar points - quadratic surface Linear 8-point algorithm Nonlinear 5-point algorithms yields up to 10 solutions CS223b

3D Structure Recovery Eliminate one of the scale’s Solve LLSE problem unknowns Eliminate one of the scale’s Solve LLSE problem Alternatively recover each point depth separately If the configuration is non-critical, the Euclidean structure of the points and motion of the camera can be reconstructed up to a universal scale. CS223b

Example Two views Point Feature Matching CS223b

Sparse Structure Recovery Example Epipolar Geometry Camera Pose and Sparse Structure Recovery CS223b

Epipolar Geometry - Planar case Plane in first camera coordinate frame Image correspondences Planar homography Linear mapping relating two corresponding planar points in two views CS223b

Decomposition of H (into motion and plane normal) Algebraic elimination of depth can be estimated linearly Normalization of Decomposition of H into 4 solutions CS223b

Uncalibrated Camera calibrated coordinates Linear transformation pixel CS223b

Uncalibrated Epipolar Geometry Epipolar constraint Fundamental matrix CS223b

Properties of the Fundamental Matrix Epipolar lines Epipoles Image correspondences CS223b

Epipolar Geometry for Parallel Cameras Epipoles are at infinite Epipolar lines are parallel to the baseline CS223b

Properties of the Fundamental Matrix A nonzero matrix is a fundamental matrix if has a singular value decomposition (SVD) with for some . There is little structure in the matrix except that CS223b

Estimating Fundamental Matrix Find such F that the epipolar error is minimized Pixel coordinates Fundamental matrix can be estimated up to scale Denote Rewrite Collect constraints from all points CS223b

Two view linear algorithm – 8-point algorithm Solve the LLSE problem: Solution eigenvector associated with smallest eigenvalue of Compute SVD of F recovered from data Project onto the essential manifold: cannot be unambiguously decomposed into pose and calibration CS223b

Projective Reconstruction Euclidean Motion Cannot be obtained in uncalibrated setting (F cannot be uniquely decomposed into R,T and K matrix) Can we still say something about 3D ? Notion of the projective 3D structure (study of projective geometry) Euclidean reconstruction Projective reconstruction CS223b

Euclidean vs Projective reconstruction Euclidean reconstruction – true metric properties of objects lenghts (distances), angles, parallelism are preserved Unchanged under rigid body transformations => Euclidean Geometry – properties of rigid bodies under rigid body transformations, similarity transformation Projective reconstruction – lengths, angles, parallelism are NOT preserved – we get distorted images of objects – their distorted 3D counterparts --> 3D projective reconstruction => Projective Geometry CS223b

Image rectification Given general displacement how to warp the views Such that epipolar lines are parallel to each other How to warp it back to canonical configuration (more details later) (Seitz) CS223b

Epipolar rectification Rectified Image Pair Corresponding epipolar lines are aligned with the scan-lines Search for dense correspondence is a 1D search CS223b

Epipolar rectification Rectified Image Pair CS223b

Special Motions – Pure Rotation Calibrated Two views related by rotation only Uncalibrated Case Mapping to a reference view – H can be estimated Mapping to a cylindrical surface - applications – image mosaics CS223b

Epipolar rectification Rectified Image Pair Corresponding epipolar lines are aligned with the scan-lines Search for dense correspondence is a 1D search CS223b

Using H p’ p Image based rectification (given some points in 3D world) compute H which would map them into a square Use H to rectify the entire image In calibrated case inter-image homography CS223b

Single view modeling in projective setting Vermeer’s Music Lesson Planes and distances from the planes can be computed Single view Rendering (A. Crimisi) CS223b

More on stereo CS223b

Usually, order of points in two images is same. Is this always true? Ordering constraint Usually, order of points in two images is same. Is this always true? If we match pixel i in image 1 to pixel j in image 2, no matches that follow will affect which are the best preceding matches. Example with pixels (a la Cox et al.). CS223b

The Ordering Constraint Points on the epipolar lines appear in the same order But it is not always the case ... This enables dynamic programming CS223b

Dynamic Programming (Baker and Binford, 1981) Find the minimum-cost path going monotonically down and right from the top-left corner of the graph to its bottom-right corner. Nodes = matched feature points (e.g., edge points). Arcs = matched intervals along the epipolar lines. Arc cost = discrepancy between intervals. CS223b

Dynamic Programming (Baker and Binford, 1981) Find the minimum-cost path going monotonically down and right from the top-left corner of the graph to its bottom-right corner. Nodes = matched feature points (e.g., edge points). Arcs = matched intervals along the epipolar lines. Arc cost = discrepancy between intervals. CS223b

Dynamic Programming (Ohta and Kanade, 1985) CS223b Reprinted from “Stereo by Intra- and Intet-Scanline Search,” by Y. Ohta and T. Kanade, IEEE Trans. on Pattern Analysis and Machine Intelligence, 7(2):139-154 (1985).  1985 IEEE.

Other constraints Smoothness: disparity usually doesn’t change too quickly. Unfortunately, this makes the problem 2D again. Solved with a host of graph algorithms, Markov Random Fields, Belief Propagation, …. Uniqueness constraint (each feature can at most have one match) Occlusion and disparity are connected. CS223b