Correspondence and Stereopsis Original notes by W. Correa. Figures from [Forsyth & Ponce] and [Trucco & Verri]

Slides:



Advertisements
Similar presentations
Lecture 11: Two-view geometry
Advertisements

Stereo Vision Reading: Chapter 11
CS 376b Introduction to Computer Vision 04 / 21 / 2008 Instructor: Michael Eckmann.
Public Library, Stereoscopic Looking Room, Chicago, by Phillips, 1923.
Gratuitous Picture US Naval Artillery Rangefinder from World War I (1918)!!
MASKS © 2004 Invitation to 3D vision Lecture 7 Step-by-Step Model Buidling.
Two-view geometry.
Lecture 8: Stereo.
Stereo.
Camera calibration and epipolar geometry
3D Computer Vision and Video Computing 3D Vision Topic 3 of Part II Stereo Vision CSc I6716 Spring 2011 Zhigang Zhu, City College of New York
Last Time Pinhole camera model, projection
Stanford CS223B Computer Vision, Winter 2005 Lecture 6: Stereo 2 Sebastian Thrun, Stanford Rick Szeliski, Microsoft Hendrik Dahlkamp and Dan Morris, Stanford.
Multiple View Geometry : Computational Photography Alexei Efros, CMU, Fall 2005 © Martin Quinn …with a lot of slides stolen from Steve Seitz and.
Epipolar geometry. (i)Correspondence geometry: Given an image point x in the first view, how does this constrain the position of the corresponding point.
Introduction to Computer Vision 3D Vision Topic 9 Stereo Vision (I) CMPSCI 591A/691A CMPSCI 570/670.
Stereopsis Mark Twain at Pool Table", no date, UCR Museum of Photography.
The plan for today Camera matrix
CS 223b 1 More on stereo and correspondence. CS 223b 2 =?f g Mostpopular For each window, match to closest window on epipolar line in other image. (slides.
3D Computer Vision and Video Computing 3D Vision Topic 4 of Part II Stereo Vision CSc I6716 Spring 2008 Zhigang Zhu, City College of New York
3D Computer Vision and Video Computing 3D Vision Lecture 15 Stereo Vision (II) CSC 59866CD Fall 2004 Zhigang Zhu, NAC 8/203A
3D Computer Vision and Video Computing 3D Vision Lecture 14 Stereo Vision (I) CSC 59866CD Fall 2004 Zhigang Zhu, NAC 8/203A
May 2004Stereo1 Introduction to Computer Vision CS / ECE 181B Tuesday, May 11, 2004  Multiple view geometry and stereo  Handout #6 available (check with.
CSE473/573 – Stereo Correspondence
Announcements PS3 Due Thursday PS4 Available today, due 4/17. Quiz 2 4/24.
Stereo Sebastian Thrun, Gary Bradski, Daniel Russakoff Stanford CS223B Computer Vision (with slides by James Rehg and.
COMP322/S2000/L271 Stereo Imaging Ref.V.S.Nalwa, A Guided Tour of Computer Vision, Addison Wesley, (ISBN ) Slides are adapted from CS641.
Multiple View Geometry : Computational Photography Alexei Efros, CMU, Fall 2006 © Martin Quinn …with a lot of slides stolen from Steve Seitz and.
Stereo matching “Stereo matching” is the correspondence problem –For a point in Image #1, where is the corresponding point in Image #2? C1C1 C2C2 ? ? C1C1.
Stereo matching Class 10 Read Chapter 7 Tsukuba dataset.
Structure Computation. How to compute the position of a point in 3- space given its image in two views and the camera matrices of those two views Use.
Stereo vision A brief introduction Máté István MSc Informatics.
3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective.
Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates.
Computer Vision Spring ,-685 Instructor: S. Narasimhan WH 5409 T-R 10:30am – 11:50am Lecture #15.
Lecture 11 Stereo Reconstruction I Lecture 11 Stereo Reconstruction I Mata kuliah: T Computer Vision Tahun: 2010.
Structure from images. Calibration Review: Pinhole Camera.
Lecture 12 Stereo Reconstruction II Lecture 12 Stereo Reconstruction II Mata kuliah: T Computer Vision Tahun: 2010.
John Morris These slides were adapted from a set of lectures written by Mircea Nicolescu, University of Nevada at Reno Stereo Vision Iolanthe in the Bay.
Epipolar geometry Epipolar Plane Baseline Epipoles Epipolar Lines
Shape from Stereo  Disparity between two images  Photogrammetry  Finding Corresponding Points Correlation based methods Feature based methods.
Stereo Vision Reading: Chapter 11 Stereo matching computes depth from two or more images Subproblems: –Calibrating camera positions. –Finding all corresponding.
Computer Vision, Robert Pless
EECS 274 Computer Vision Stereopsis.
Computer Vision Stereo Vision. Bahadir K. Gunturk2 Pinhole Camera.
Computer Vision Lecture #10 Hossam Abdelmunim 1 & Aly A. Farag 2 1 Computer & Systems Engineering Department, Ain Shams University, Cairo, Egypt 2 Electerical.
CSE 185 Introduction to Computer Vision Stereo. Taken at the same time or sequential in time stereo vision structure from motion optical flow Multiple.
Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform  Location of the impulse.
Two-view geometry. Epipolar Plane – plane containing baseline (1D family) Epipoles = intersections of baseline with image planes = projections of the.
stereo Outline : Remind class of 3d geometry Introduction
Feature Matching. Feature Space Outlier Rejection.
Solving for Stereo Correspondence Many slides drawn from Lana Lazebnik, UIUC.
Computer vision: models, learning and inference M Ahad Multiple Cameras
John Morris Stereo Vision (continued) Iolanthe returns to the Waitemata Harbour.
John Morris These slides were adapted from a set of lectures written by Mircea Nicolescu, University of Nevada at Reno Stereo Vision Iolanthe in the Bay.
Correspondence and Stereopsis. Introduction Disparity – Informally: difference between two pictures – Allows us to gain a strong sense of depth Stereopsis.
CSE 185 Introduction to Computer Vision Stereo 2.
CS 6501: 3D Reconstruction and Understanding Stereo Cameras
55:148 Digital Image Processing Chapter 11 3D Vision, Geometry
Stereo CSE 455 Linda Shapiro.
CS4670 / 5670: Computer Vision Kavita Bala Lec 27: Stereo.
STEREOPSIS The Stereopsis Problem: Fusion and Reconstruction
Epipolar geometry.
Geometry 3: Stereo Reconstruction
EECS 274 Computer Vision Stereopsis.
Common Classification Tasks
Thanks to Richard Szeliski and George Bebis for the use of some slides
What have we learned so far?
Course 6 Stereo.
Chapter 11: Stereopsis Stereopsis: Fusing the pictures taken by two cameras and exploiting the difference (or disparity) between them to obtain the depth.
Presentation transcript:

Correspondence and Stereopsis Original notes by W. Correa. Figures from [Forsyth & Ponce] and [Trucco & Verri]

Introduction Disparity:Disparity: – Informally: difference between two pictures – Allows us to gain a strong sense of depth Stereopsis:Stereopsis: – Ability to perceive depth from disparity Goal:Goal: – Design algorithms that mimic stereopsis

Stereo Vision Two partsTwo parts – Binocular fusion of features observed by the eyes – Reconstruction of their three-dimensional preimage

Stereo Vision – Easy Case 1 single point being observed1 single point being observed – The preimage can be found at the intersection of the rays from the focal points to the image points

Stereo Vision – Hard Case Many points being observedMany points being observed – Need some method to establish correspondences

Components of Stereo Vision Systems Camera calibration: previous lectureCamera calibration: previous lecture Image rectification: simplifies the search for correspondencesImage rectification: simplifies the search for correspondences Correspondence: which item in the left image corresponds to which item in the right imageCorrespondence: which item in the left image corresponds to which item in the right image Reconstruction: recovers 3-D information from the 2-D correspondencesReconstruction: recovers 3-D information from the 2-D correspondences

Epipolar Geometry Epipolar constraint: corresponding points must lie on conjugate epipolar linesEpipolar constraint: corresponding points must lie on conjugate epipolar lines – Search for correspondences becomes a 1-D problem

Image Rectification Warp images such that conjugate epipolar lines become collinear and parallel to u axisWarp images such that conjugate epipolar lines become collinear and parallel to u axis

Image Rectification (cont.) Perform by rotating the camerasPerform by rotating the cameras Not equivalent to rotating the images Not equivalent to rotating the images The lines through the centers become parallel to each other, and the epipoles move to infinityThe lines through the centers become parallel to each other, and the epipoles move to infinity

Image Rectification (cont.) Given extrinsic parameters T and R (relative position and orientation of the two cameras)Given extrinsic parameters T and R (relative position and orientation of the two cameras) – Rotate the left camera about the projection center so that the the epipolar lines become parallel to the horizontal axis – Apply the same rotation to the right camera – Rotate the right camera by R – Adjust the scale in both camera reference frames

Disparity With rectified images, disparity is just (horizontal) displacement of corresponding features in the two imagesWith rectified images, disparity is just (horizontal) displacement of corresponding features in the two images – Disparity = 0 for distant points – Larger disparity for closer points – Depth of point proportional to 1/disparity

Correspondence Given an element in the left image, find the corresponding element in the right imageGiven an element in the left image, find the corresponding element in the right image Classes of methodsClasses of methods – Correlation-based – Feature-based

Correlation-Based Correspondence Input: rectified stereo pair and a point (u,v) in the first imageInput: rectified stereo pair and a point (u,v) in the first image Method:Method: – Form window of size (2m+1)  (2n+1) centered at (u,v) and assemble points into the vector w – For each potential match (u+d,v) in the second image, compute w' and the normalized correlation between w and w‘

Sum of Squared Differences Recall: SSD for image similarityRecall: SSD for image similarity Negative sign so that higher values mean greater similarityNegative sign so that higher values mean greater similarity

Normalized Cross-Correlation Normalize to eliminate brightness sensitivity: whereNormalize to eliminate brightness sensitivity: where Helps for non-diffuse scenes, can hurt for perfectly diffuse onesHelps for non-diffuse scenes, can hurt for perfectly diffuse ones

Correlation-Based Correspondence (cont.) Main problem:Main problem: – Assumes that the observed surface is locally parallel to the two image planes – If not, unequal amounts of foreshortening in images – Alleviate by computing initial disparity, warping the images, iterating Other problems:Other problems: – Not robust against noise – Similar pixels may not correspond to physical features

Feature-Based Correspondence Main idea: physically-significant features should be preferred to matches between raw pixel intensitiesMain idea: physically-significant features should be preferred to matches between raw pixel intensities Instead of correlation-like measures, use a measure of the distance between feature descriptorsInstead of correlation-like measures, use a measure of the distance between feature descriptors Typical features: points, lines, and cornersTypical features: points, lines, and corners Example: Marr-Poggio-Grimson algorithmExample: Marr-Poggio-Grimson algorithm

Marr-Poggio-Grimson Algorithm Convolve images with Laplacian of Gaussian filters with decreasing widthsConvolve images with Laplacian of Gaussian filters with decreasing widths Find zero crossings of the Laplacian along horizontal scanlines of the filtered imagesFind zero crossings of the Laplacian along horizontal scanlines of the filtered images For each , match zero crossings with same parity and similar orientations in a [– w .. w  ] disparity range, withFor each , match zero crossings with same parity and similar orientations in a [– w .. w  ] disparity range, with

Marr-Poggio-Grimson Algorithm (cont.) Use disparities found at larger scales to control eye vergence and cause unmatched regions at smaller scales to come into correspondenceUse disparities found at larger scales to control eye vergence and cause unmatched regions at smaller scales to come into correspondence

Marr-Poggio-Grimson Algorithm (cont.)

Ordering Constraint Order of matching features usually the same in both imagesOrder of matching features usually the same in both images But not always: occlusionBut not always: occlusion

Dynamic Programming Treat feature correspondence as graph problemTreat feature correspondence as graph problem Left image features Right image features Cost of edges = similarity of regions between image features

Dynamic Programming Find min-cost path through graphFind min-cost path through graph Left image features Right image features

Reconstruction Given pair of image points p and p', and focal points O and O', find preimage PGiven pair of image points p and p', and focal points O and O', find preimage P In theory: find P by intersecting the rays R=Op and R'=Op'In theory: find P by intersecting the rays R=Op and R'=Op' In practice: R and R' won't actually intersect due to calibration and feature localization errorsIn practice: R and R' won't actually intersect due to calibration and feature localization errors

Reconstruction Approaches GeometricGeometric – Construct the line segment perpendicular to R and R' that intersects both rays and take its mid-point

Reconstruction Approaches Image-space: find the point P whose projection onto the images minimizes distance to desired correspondencesImage-space: find the point P whose projection onto the images minimizes distance to desired correspondences Nonlinear optimizationNonlinear optimization