Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens Stereo

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Summary Stereo n Epipolar Geometry n Fundamental/Essential Matrix p l p r P OlOl OrOr elel erer PlPl PrPr Epipolar Plane Epipolar Lines Epipoles

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Correspondence: Where to search? Search window?

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Stereo Vision 2: Outline n Image Rectification n Correspondence n Active Stereo n Dense and Layered Stereo n Smoothing With Markov Random Fields

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Rectification n Problem: Epipolar lines not parallel to scan lines p l p r P OlOl OrOr elel erer PlPl PrPr Epipolar Plane Epipolar Lines Epipoles

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Rectification n Problem: Epipolar lines not parallel to scan lines p l p r P OlOl OrOr PlPl PrPr Epipolar Plane Epipolar Lines Epipoles at ininity Rectified Images

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Epipolar Rectified Stereo Images Epipolar line

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Epipolar Rectified Images Source: A. Fusiello, Verona, 2000]

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Example Rectification

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Final Step: Image Normalization n Even when the cameras are identical models, there can be differences in gain and sensitivity. n The cameras do not see exactly the same surfaces, so their overall light levels can differ. n For these reasons and more, it is a good idea to normalize the pixels in each window:

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Stereo Vision 2: Outline n Image Rectification n Correspondence n Active Stereo n Dense and Layered Stereo n Smoothing With Markov Random Fields

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Correspondence Phantom points

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Correspondence via Correlation Rectified images LeftRight scanline SSD error disparity (Same as max-correlation / max-cosine for normalized image patch)

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Images as Vectors LeftRight Each window is a vector in an m 2 dimensional vector space. Normalization makes them unit length.

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Image Metrics (Normalized) Sum of Squared Differences Normalized Correlation

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Correspondence Using Correlation LeftDisparity Map Images courtesy of Point Grey Research

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 LEFT IMAGE corner line structure Correspondence By Features

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Correspondence By Features RIGHT IMAGE corner line structure n Search in the right image… the disparity (dx, dy) is the displacement when the similarity measure is maximum

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Stereo Correspondences …… Left scan lineRight scan line

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Stereo Correspondences …… Left scanlineRight scanline Match OcclusionDisocclusion

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Search Over Correspondences Three cases: –Sequential – cost of match –Occluded – cost of no match –Disoccluded – cost of no match Left scanline Right scanline Occluded Pixels Disoccluded Pixels

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path. Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Terminal Stereo Matching with Dynamic Programming

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 DP Algorithm: V[0,0] = 0 V[i,k] = min { V[i-1,k-1] + m(i,j), c+V[i, k-1], c+V[i-1,k] } d[i,k] = argmin { … } Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Terminal Stereo Matching with Dynamic Programming

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Terminal Stereo Matching with Dynamic Programming DP Algorithm: V[0,0] = 0 V[i,k] = min { V[i-1,k-1] + m(i,j), c+V[i, k-1], c+V[i-1,k] } d[i,k] = argmin { … }

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Find Stereo Alignment D=[X,Y] repeat until D=[1,1] add D to alignment D = d[D] end Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Terminal Stereo Matching with Dynamic Programming

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Commercial-Grade Stereo n Tyzx, a leading stereo camera manufacturer n (here strapped on our DARPA Grand Challenge vehicle) Disparity map

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Dense Stereo Matching: Examples n View extrapolation results input depth image novel view [Matthies,Szeliski,Kanade’88]

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Dense Stereo Matching n Some other view extrapolation results inputdepth imagenovel view

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Dense Stereo Matching n Compute certainty map from correlations input depth map certainty map

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 DP for Correspondence n Does this always work? n When would it fail? –Failure Example 1 –Failure Example 2 –Failure Example 3 –Failure Example 4

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Correspondence Problem 1 n Ambiguities

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Correspondence Problem 2 n Multiple occluding objects Figure from Forsyth & Ponce

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Correspondence Problem 3 n Correspondence fail for smooth surfaces (edge = occlusion boundary, poorly localized) n There is currently no good solution to this correspondence problem

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Correspondence Problem 4 n Regions without texture n Highly Specular surfaces n Translucent objects

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Some Stereo Results Side view Top view

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 More Stereo Results Side view

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 A True Challenge!

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Stereo Vision 2: Outline n Image Rectification n Correspondence n Active Stereo n Dense and Layered Stereo n Smoothing With Markov Random Fields

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 How can We Improve Stereo? Space-time stereo scanner uses unstructured light to aid in correspondence Result: Dense 3D mesh (noisy)

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Active Stereo: Adding Texture to Scene By James Davis, Honda Research, Now UCSC

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 rectified Active Stereo (Structured Light)

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Structured Light: 3-D Result 3D Snapshot By James Davis, Honda Research

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Time of Flight Sensor: Shutter

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Time of Flight Sensor: Shutter

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Time of Flight Sensor: Shutter

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Time of Flight Sensor: Shutter

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Time of Flight Sensor: Shutter

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Scanning Laser Range Finders

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Scanning Laser Results

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Scanning Laser Results

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Stereo Vision 2: Outline n Image Rectification n Correspondence n Active Stereo n Dense and Layered Stereo n Smoothing With Markov Random Fields

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Layered Stereo n Assign pixel to different “layers” (objects, sprites)

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Layered Stereo n Track each layer from frame to frame, compute plane eqn. and composite mosaic n Re-compute pixel assignment by comparing original images to sprites

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Layered Stereo n Re-synthesize original or novel images from collection of sprites

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Layered Stereo n Advantages: –can represent occluded regions –can represent transparent and border (mixed) pixels (sprites have alpha value per pixel) –works on texture-less interior regions n Limitations: –fails for high depth-complexity scenes

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Fitting Planar Surfaces (with EM) ** ****

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Expectation Maximization n 3D Model: Planar surface in 3D Distance point-surface surface normal a y x z displacement b

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Mixture Measurement Model Case 1: Measurement z i caused by plane  j §Case 2: Measurement z i caused by something else

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Measurement Model with Correspondences correspondence variables C : }

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Expected Log-Likelihood Function …after some simple math mapping with known data association probabilistic data association

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 The EM Algorithm n E-step: given plane params, compute n M-step: given expectations, compute

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Choosing the “Right” Number of Planes: AIC J=2J=3J=5J=0J=1J=4 increased data likelihoodincreased prior probability

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Determining Number of Surfaces J =1 First model component * * J =1 E-Step * * J =3 Add model components J =3 E-Step J =3 M-step J =1 Prune model J =3 Add model components J =3 E/M Steps * J =2 Prune model

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Layered Stereo n Resulting sprite collection

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Layered Stereo n Estimated depth map

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Example (here with laser range finder)

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Example (here with laser range finder) n Another Example

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Stereo Vision 2: Outline n Image Rectification n Correspondence n Active Stereo n Dense and Layered Stereo n Smoothing With Markov Random Fields

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Motivation and Goals James Diebel

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Motivation and Goals James Diebel

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Network of Constraints (Markov Random Field) James Diebel

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 MRF Approach to Smoothing n Potential function: contains a sensor-model term and a surface prior n The edge potential is important! n Minimize  by conjugate gradient –Optimize systems with tens of thousands of parameters in just a couple seconds –Time to converge is O(N), between 0.7 sec (25,000 nodes in the MRF) and 25 sec (900,000 nodes) Diebel/Thrun, 2006

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Possible Edge Potential Functions

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Results: Smoothing James Diebel

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Results: Smoothing James Diebel

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Results: Smoothing James Diebel

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Results: Smoothing James Diebel

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Movies… Movies in Windows Media Player

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Summary n Image Rectification n Correspondence n Active Stereo n Dense and Layered Stereo n Smoothing With Markov Random Fields