Stanford CS223B Computer Vision, Winter 2006 Lecture 6 Stereo II Professor Sebastian Thrun CAs: Dan Maynes-Aminzade, Mitul Saha, Greg Corrado Stereo

Sebastian Thrun Stanford University CS223B Computer Vision Stereo Vision: Outline n Basic Equations n Epipolar Geometry n Image Rectification n Reconstruction n Correspondence n Active Range Imaging Technology n Dense and Layered Stereo n Smoothing With Markov Random Fields

Sebastian Thrun Stanford University CS223B Computer Vision A Last Word on Preprocessing….

Sebastian Thrun Stanford University CS223B Computer Vision Epipolar Rectified Images Epipolar line

Sebastian Thrun Stanford University CS223B Computer Vision Epipolar Rectified Images Source: A. Fusiello, Verona, 2000]

Sebastian Thrun Stanford University CS223B Computer Vision Image Normalization n Even when the cameras are identical models, there can be differences in gain and sensitivity. n The cameras do not see exactly the same surfaces, so their overall light levels can differ. n For these reasons and more, it is a good idea to normalize the pixels in each window:

Sebastian Thrun Stanford University CS223B Computer Vision Stereo Vision: Outline n Basic Equations n Epipolar Geometry n Image Rectification n Reconstruction n Correspondence n Active Range Imaging Technology n Dense and Layered Stereo n Smoothing With Markov Random Fields

Sebastian Thrun Stanford University CS223B Computer Vision Correspondence Phantom points

Sebastian Thrun Stanford University CS223B Computer Vision Correspondence via Correlation Rectified images LeftRight scanline SSD error disparity (Same as max-correlation / max-cosine for normalized image patch)

Sebastian Thrun Stanford University CS223B Computer Vision Images as Vectors LeftRight Each window is a vector in an m 2 dimensional vector space. Normalization makes them unit length.

Sebastian Thrun Stanford University CS223B Computer Vision Image Metrics (Normalized) Sum of Squared Differences Normalized Correlation

Sebastian Thrun Stanford University CS223B Computer Vision Correspondence Using Correlation LeftDisparity Map Images courtesy of Point Grey Research

Sebastian Thrun Stanford University CS223B Computer Vision LEFT IMAGE corner line structure Correspondence By Features

Sebastian Thrun Stanford University CS223B Computer Vision Correspondence By Features RIGHT IMAGE corner line structure n Search in the right image… the disparity (dx, dy) is the displacement when the similarity measure is maximum

Sebastian Thrun Stanford University CS223B Computer Vision Stereo Correspondences …… Left scanlineRight scanline

Sebastian Thrun Stanford University CS223B Computer Vision Stereo Correspondences …… Left scanlineRight scanline Match OcclusionDisocclusion

Sebastian Thrun Stanford University CS223B Computer Vision Search Over Correspondences Three cases: –Sequential – cost of match –Occluded – cost of no match –Disoccluded – cost of no match Left scanline Right scanline Occluded Pixels Disoccluded Pixels

Sebastian Thrun Stanford University CS223B Computer Vision Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path. Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Terminal Stereo Matching with Dynamic Programming

Sebastian Thrun Stanford University CS223B Computer Vision Stereo Matching with Dynamic Programming Dynamic programming yields the optimal path through grid. This is the best set of matches that satisfy the ordering constraint Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Start End

Sebastian Thrun Stanford University CS223B Computer Vision Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path. Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Terminal Stereo Matching with Dynamic Programming

Sebastian Thrun Stanford University CS223B Computer Vision Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path. Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Terminal Stereo Matching with Dynamic Programming

Sebastian Thrun Stanford University CS223B Computer Vision Dense Stereo Matching: Examples n View extrapolation results input depth image novel view [Matthies,Szeliski,Kanade’88]

Sebastian Thrun Stanford University CS223B Computer Vision Dense Stereo Matching n Some other view extrapolation results inputdepth imagenovel view

Sebastian Thrun Stanford University CS223B Computer Vision Dense Stereo Matching n Compute certainty map from correlations input depth map certainty map

Sebastian Thrun Stanford University CS223B Computer Vision DP for Correspondence n Does this always work? n When would it fail? –Failure Example 1 –Failure Example 2 –Failure Example 3

Sebastian Thrun Stanford University CS223B Computer Vision Correspondence Problem 1 n It is fundamentally ambiguous, even with stereo constraints Ordering constraint……and its failure Figure from Forsyth & Ponce

Sebastian Thrun Stanford University CS223B Computer Vision Correspondence Problem 2 n Correspondence fail for smooth surfaces n There is currently no good solution to the correspondence problem

Sebastian Thrun Stanford University CS223B Computer Vision Correspondence Problem 3 n Regions without texture n Highly Specular surfaces n Translucent objects

Sebastian Thrun Stanford University CS223B Computer Vision Stereo Vision: Outline n Basic Equations n Epipolar Geometry n Image Rectification n Reconstruction n Correspondence n Active Range Imaging Technology n Dense and Layered Stereo n Smoothing With Markov Random Fields

Sebastian Thrun Stanford University CS223B Computer Vision How can We Improve Stereo? Space-time stereo scanner uses unstructured light to aid in correspondence Result: Dense 3D mesh (noisy)

Sebastian Thrun Stanford University CS223B Computer Vision Prof Marc Stanford By James Davis, Honda Research, Now UCSC

Sebastian Thrun Stanford University CS223B Computer Vision rectified Active Stereo (Structured Light)

Sebastian Thrun Stanford University CS223B Computer Vision Structured Light: 3-D Result 3D Model3D Snapshot By James Davis, Honda Research

Sebastian Thrun Stanford University CS223B Computer Vision Time of Flight Sensor: Shutter

Sebastian Thrun Stanford University CS223B Computer Vision Time of Flight Sensor: Shutter

Sebastian Thrun Stanford University CS223B Computer Vision Time of Flight Sensor: Shutter

Sebastian Thrun Stanford University CS223B Computer Vision Stereo Vision: Outline n Basic Equations n Epipolar Geometry n Image Rectification n Reconstruction n Correspondence n Active Range Imaging Technology n Layered Stereo n Smoothing With Markov Random Fields

Sebastian Thrun Stanford University CS223B Computer Vision Disclaimer n The Following Material Shall Not Be Required For the Midterm Exam

Sebastian Thrun Stanford University CS223B Computer Vision Layered Stereo n Assign pixel to different “layers” (objects, sprites)

Sebastian Thrun Stanford University CS223B Computer Vision Layered Stereo n Track each layer from frame to frame, compute plane eqn. and composite mosaic n Re-compute pixel assignment by comparing original images to sprites

Sebastian Thrun Stanford University CS223B Computer Vision Layered Stereo n Re-synthesize original or novel images from collection of sprites

Sebastian Thrun Stanford University CS223B Computer Vision Layered Stereo n Advantages: –can represent occluded regions –can represent transparent and border (mixed) pixels (sprites have alpha value per pixel) –works on texture-less interior regions n Limitations: –fails for high depth-complexity scenes

Sebastian Thrun Stanford University CS223B Computer Vision Fitting Planar Surfaces (with EM) ** ****

Sebastian Thrun Stanford University CS223B Computer Vision Expectation Maximization n 3D Model: Planar surface in 3D Distance point-surface surface normal  y x z displacement 

Sebastian Thrun Stanford University CS223B Computer Vision Mixture Measurement Model Case 1: Measurement z i caused by plane  j  Case 2: Measurement z i caused by something else

Sebastian Thrun Stanford University CS223B Computer Vision Measurement Model with Correspondences correspondence variables C : }

Sebastian Thrun Stanford University CS223B Computer Vision Expected Log-Likelihood Function …after some simple math mapping with known data association probabilistic data association

Sebastian Thrun Stanford University CS223B Computer Vision The EM Algorithm n E-step: given plane params, compute n M-step: given expectations, compute

Sebastian Thrun Stanford University CS223B Computer Vision Choosing the “Right” Number of Planes: AIC J=2J=3J=5J=0J=1J=4 increased data likelihoodincreased prior probability

Sebastian Thrun Stanford University CS223B Computer Vision Determining Number of Surfaces J =1 First model component * * J =1 E-Step * * J =3 Add model components J =3 E-Step J =3 M-step J =1 Prune model J =3 Add model components J =3 E/M Steps * J =2 Prune model

Sebastian Thrun Stanford University CS223B Computer Vision Layered Stereo n Resulting sprite collection

Sebastian Thrun Stanford University CS223B Computer Vision Layered Stereo n Estimated depth map

Sebastian Thrun Stanford University CS223B Computer Vision Stereo Vision: Outline n Basic Equations n Epipolar Geometry n Image Rectification n Reconstruction n Correspondence n Active Range Imaging Technology n Dense and Layered Stereo n Smoothing With Markov Random Fields

Sebastian Thrun Stanford University CS223B Computer Vision Motivation and Goals James Diebel

Sebastian Thrun Stanford University CS223B Computer Vision Motivation and Goals James Diebel

Sebastian Thrun Stanford University CS223B Computer Vision Network of Constraints (Markov Random Field) James Diebel

Sebastian Thrun Stanford University CS223B Computer Vision MRF Approach to Smoothing n Potential function: contains a sensor-model term and a surface prior n The edge potential is important! n Minimize  by conjugate gradient –Optimize systems with tens of thousands of parameters in just a couple seconds –Time to converge is O(N), between 0.7 sec (25,000 nodes in the MRF) and 25 sec (900,000 nodes) Diebel/Thrun, 2006

Sebastian Thrun Stanford University CS223B Computer Vision Possible Edge Potential Functions

Sebastian Thrun Stanford University CS223B Computer Vision Results: Smoothing James Diebel

Sebastian Thrun Stanford University CS223B Computer Vision Results: Smoothing James Diebel

Sebastian Thrun Stanford University CS223B Computer Vision Results: Smoothing James Diebel

Sebastian Thrun Stanford University CS223B Computer Vision Results: Smoothing James Diebel

Sebastian Thrun Stanford University CS223B Computer Vision Movies… Movies in Windows Media Player

Sebastian Thrun Stanford University CS223B Computer Vision Stereo Vision: Outline n Basic Equations n Epipolar Geometry n Image Rectification n Reconstruction n Correspondence n Active Range Imaging Technology n Dense and Layered Stereo n Smoothing With Markov Random Fields