Stereo Sebastian Thrun, Gary Bradski, Daniel Russakoff Stanford CS223B Computer Vision (with slides by James Rehg and Zhigang Zhu) Stereo

Sebastian Thrun Stanford University CS223B Computer Vision Stereo Vision: Illustration

Sebastian Thrun Stanford University CS223B Computer Vision Stereo Vision: Outline n Basic Equations n Epipolar Geometry n Image Rectification n Reconstruction n Correspondence n (Active Range Imaging Techniques)

Sebastian Thrun Stanford University CS223B Computer Vision Pinhole Camera Model Image plane Focal length f Center of projection

Sebastian Thrun Stanford University CS223B Computer Vision Pinhole Camera Model Image plane

Sebastian Thrun Stanford University CS223B Computer Vision Pinhole Camera Model Image plane

Sebastian Thrun Stanford University CS223B Computer Vision Basic Stereo Derivations

Sebastian Thrun Stanford University CS223B Computer Vision Basic Stereo Derivations

Sebastian Thrun Stanford University CS223B Computer Vision What If…?

Sebastian Thrun Stanford University CS223B Computer Vision Epipolar Geometry p l p r P OlOl OrOr XlXl XrXr PlPl PrPr flfl frfr ZlZl YlYl ZrZr YrYr

Sebastian Thrun Stanford University CS223B Computer Vision Epipolar Geometry p l p r P OlOl OrOr elel erer PlPl PrPr Epipolar Plane Epipolar Lines Epipoles

Sebastian Thrun Stanford University CS223B Computer Vision Epipolar Geometry n Epipolar plane: plane going through point P and the centers of projection (COPs) of the two cameras n Epipoles: The image in one camera of the COP of the other n Epipolar Constraint: Corresponding points must lie on epipolar lines

Sebastian Thrun Stanford University CS223B Computer Vision Essential Matrix p l p r P OlOl OrOr elel erer PlPl PrPr Orthogonality T, P l, P l  T : Coordinate Transformation: Resolves to Essential Matrix

Sebastian Thrun Stanford University CS223B Computer Vision Essential Matrix p l p r P OlOl OrOr elel erer PlPl PrPr Projective Line:

Sebastian Thrun Stanford University CS223B Computer Vision Fundamental Matrix n Same as Essential Matrix in Camera Pixel Coordinates Pixel coordinates Intrinsic parameters

Sebastian Thrun Stanford University CS223B Computer Vision Computing F: The Eight-Point Algorithm n Input: n point correspondences ( n >= 8) –Construct homogeneous system Ax= 0 from x = (f 11,f 12,,f 13, f 21,f 22,f 23 f 31,f 32, f 33 ) : entries in F Each correspondence give one equation A is a nx9 matrix –Obtain estimate F^ by SVD of A: x (up to a scale) is column of V corresponding to the least singular value –Enforce singularity constraint: since Rank (F) = 2 Compute SVD of F: Set the smallest singular value to 0: D -> D’ Correct estimate of F : n Output: the estimate of the fundamental matrix F’ n Similarly we can compute E given intrinsic parameters

Sebastian Thrun Stanford University CS223B Computer Vision Recitification Idea: Align Epipolar Lines with Scan Lines. n Question: What type transformation?

Sebastian Thrun Stanford University CS223B Computer Vision Locating the Epipoles p l p r P OlOl OrOr elel erer PlPl PrPr n Input: Fundamental Matrix F –Find the SVD of F –The epipole e l is the column of V corresponding to the null singular value (as shown above) –The epipole e r is the column of U corresponding to the null singular value (similar treatment as for e l ) n Output: Epipole e l and e r e l lies on all the epipolar lines of the left image

Sebastian Thrun Stanford University CS223B Computer Vision Stereo Rectification (see Trucco) Stereo System with Parallel Optical Axes n Epipoles are at infinity n Horizontal epipolar lines p l p r P OlOl OrOr XlXl XrXr PlPl PrPr ZlZl YlYl ZrZr YrYr T

Sebastian Thrun Stanford University CS223B Computer Vision p l p r P OlOl OrOr PlPl PrPr Reconstruction (3-D): Idealized

Sebastian Thrun Stanford University CS223B Computer Vision p l p r P OlOl OrOr PlPl PrPr Reconstruction (3-D): Real See Trucco/Verri, pages

Sebastian Thrun Stanford University CS223B Computer Vision Correspondence Phantom points

Sebastian Thrun Stanford University CS223B Computer Vision Correspondence via Correlation Rectified images LeftRight scanline SSD error disparity (Same as max-correlation / max-cosine for normalized image patch)

Sebastian Thrun Stanford University CS223B Computer Vision Image Normalization n Even when the cameras are identical models, there can be differences in gain and sensitivity. n The cameras do not see exactly the same surfaces, so their overall light levels can differ. n For these reasons and more, it is a good idea to normalize the pixels in each window:

Sebastian Thrun Stanford University CS223B Computer Vision Images as Vectors LeftRight Each window is a vector in an m 2 dimensional vector space. Normalization makes them unit length.

Sebastian Thrun Stanford University CS223B Computer Vision Image Metrics (Normalized) Sum of Squared Differences Normalized Correlation

Sebastian Thrun Stanford University CS223B Computer Vision Correspondence Using Correlation LeftDisparity Map Images courtesy of Point Grey Research

Sebastian Thrun Stanford University CS223B Computer Vision LEFT IMAGE corner line structure Correspondence By Features

Sebastian Thrun Stanford University CS223B Computer Vision Correspondence By Features RIGHT IMAGE corner line structure n Search in the right image… the disparity (dx, dy) is the displacement when the similarity measure is maximum

Sebastian Thrun Stanford University CS223B Computer Vision Stereo Correspondences …… Left scanlineRight scanline

Sebastian Thrun Stanford University CS223B Computer Vision Stereo Correspondences …… Left scanlineRight scanline Match OcclusionDisocclusion

Sebastian Thrun Stanford University CS223B Computer Vision Search Over Correspondences Three cases: –Sequential – cost of match –Occluded – cost of no match –Disoccluded – cost of no match Left scanline Right scanline Occluded Pixels Disoccluded Pixels

Sebastian Thrun Stanford University CS223B Computer Vision Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path. Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Terminal Stereo Matching with Dynamic Programming

Sebastian Thrun Stanford University CS223B Computer Vision Stereo Matching with Dynamic Programming Dynamic programming yields the optimal path through grid. This is the best set of matches that satisfy the ordering constraint Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Start End

Sebastian Thrun Stanford University CS223B Computer Vision Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path. Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Terminal Stereo Matching with Dynamic Programming

Sebastian Thrun Stanford University CS223B Computer Vision Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path. Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Terminal Stereo Matching with Dynamic Programming

Sebastian Thrun Stanford University CS223B Computer Vision Correspondence n It is fundamentally ambiguous, even with stereo constraints Ordering constraint……and its failure Figure from Forsyth & Ponce

Sebastian Thrun Stanford University CS223B Computer Vision A Last Word on Correspondences n Correspondens fail for smooth surfaces n There is currently no good solution to the correspondence problem

Sebastian Thrun Stanford University CS223B Computer Vision Summary Stereo Vision n Epipolar Geometry: Corresponding points lie on epipolar line n Essential/Fundamental matrix: Defines this line n Eight-Point Algorithm: Recovers Fundamental matrix n Rectification: Epipolar lines parallel to scanlines n Reconstruction: Minimize quadratic distance n Correspondence: –Minimize Sum of Squares over image correlation –Minimize Sum of Squares of feature characteristics n Many correspondences: Dynamic programming along scanlines (but can fail)

Sebastian Thrun Stanford University CS223B Computer Vision How can We Improve Stereo? By James Davis, Honda Research

Sebastian Thrun Stanford University CS223B Computer Vision rectified Active Stereo (Structured Light)

Sebastian Thrun Stanford University CS223B Computer Vision Structured Light: 3-D Result 3D Model3D Snapshot By James Davis, Honda Research

Sebastian Thrun Stanford University CS223B Computer Vision Time of Flight Sensor: Shutter

Sebastian Thrun Stanford University CS223B Computer Vision Time of Flight Sensor: Shutter

Sebastian Thrun Stanford University CS223B Computer Vision Time of Flight Sensor: Shutter

Sebastian Thrun Stanford University CS223B Computer Vision Time of Flight Sensor: Scanning

Sebastian Thrun Stanford University CS223B Computer Vision Time of Flight Sensor: Scanning

Sebastian Thrun Stanford University CS223B Computer Vision Time of Flight Sensor: Scanning Cleaned up…Raw data