CSc D Computer Vision / Ioannis Stamos 3-D Computer Vision CSc Stereo

CSc D Computer Vision / Ioannis Stamos Stereopsis  Recovering 3D information (depth) from two images.  The correspondence problem.  The reconstruction problem.  Epipolar constraint.  The 8-point algorithm.

CSc D Computer Vision / Ioannis Stamos The 2 problems of Stereo  Correspondence: Which parts of the left and right images are projections of the same scene element?  Reconstruction: Given:  A number of corresponding points between the left and right image,  Information on the geometry of the stereo system, Find: Find: 3-D structure of observed objects. 3-D structure of observed objects. The setting: Simultaneous acquisition of 2 images (left, right) of a static scene.

CSc D Computer Vision / Ioannis Stamos Stereo Vision depth map

CSc D Computer Vision / Ioannis Stamos A simple stereo system f T Z P OlOl OrOr Left CameraRight Camera X-axis Z-axis clcl crcr Fixation Point:Infinity. Parallel optical axes.

CSc D Computer Vision / Ioannis StamosTriangulation f T Z P plpl prpr OlOl OrOr Left CameraRight Camera X-axis Z-axis clcl crcr Fixation Point:Infinity. Parallel optical axes. Calibrated Cameras

Triangulation f T Z P plpl prpr OlOl OrOr Left CameraRight Camera X-axis Z-axis clcl crcr Fixation Point:Infinity. Parallel optical axes. Calibrated Cameras xlxl xr Similar triangles: d:disparity (difference in retinal positions). T:baseline. Depth (Z) is inversely proportional to d (fixation at infinity)

Triangulation f T Z P plpl prpr OlOl OrOr Left CameraRight Camera X-axis Z-axis clcl crcr Fixation Point:Infinity. Parallel optical axes. Calibrated Cameras xlxl xr Similar triangles: d:disparity (difference in retinal positions). T:baseline. Baseline T: accuracy/robustness of depth calculation.

Triangulation f T Z P plpl prpr OlOl OrOr Left CameraRight Camera X-axis Z-axis clcl crcr Fixation Point:Infinity. Parallel optical axes. Calibrated Cameras xlxl xr Similar triangles: d:disparity (difference in retinal positions). T:baseline. Small baselines: less accurate measurements.

Triangulation f T Z P plpl prpr OlOl OrOr Left CameraRight Camera X-axis Z-axis clcl crcr Fixation Point:Infinity. Parallel optical axes. Calibrated Cameras xlxl xr Similar triangles: d:disparity (difference in retinal positions). T:baseline. Large baselines: occlusions/foreshortening.

CSc D Computer Vision / Ioannis Stamos Parameters of Stereo System f T Z P plpl prpr OlOl OrOr Left CameraRight Camera X-axis Z-axis clcl crcr Fixation Point:Infinity. Parallel optical axes. 1)Intrinsic parameters (i.e. f, cl, cr) 2)Extrinsic parameters: relative position and orientation of the 2 cameras. xlxl xr STEREO CALIBRATION PROBLEM

Difficulties – ambiguities, large changes of appearance, due to change Of viewpoint, non-uniquess Stereo – Photometric Constraint Stereo – Photometric Constraint Same world point has same intensity in both images. Lambertian fronto-parallel Issues (noise, specularities, foreshortening) From Jana Kosecka

CSc D Computer Vision / Ioannis Stamos Correspondence Is Difficult Ambiguity: there may be many possible 3D reconstructions.

CSc D Computer Vision / Ioannis Stamos Correspondence Is Difficult No texture: difficult to find a unique match.

CSc D Computer Vision / Ioannis Stamos Correspondence Is Difficult Foreshortening: the projection in each image is different.

Correspondence Is Difficult Occlusions: there may not be a correspondence. Assumptions: 1) Most scene points are visible from both views. 2) Corresponding image regions are similar.

CSc D Computer Vision / Ioannis Stamos Correspondence Is Difficult Curved surfaces: triangulation produces incorrect position.

CSc D Computer Vision / Ioannis Stamos Correspondence is difficult: The Ordering Constraint But it is not always the case... Points appear in the same order

CSc D Computer Vision / Ioannis Stamos More Correspondence Problems  Regions without texture  Highly Specular surfaces  Translucent objects

CSc D Computer Vision / Ioannis Stamos Methods For Correspondence  Correlation based (dense correspondences).  Feature based (such as edges/lines/corners).

Correlation-Based Methods plpl R(p l ) Left ImageRight Image 1) For each pixel pl in the left image search in a region R(pl) in the right image for corresponding pixel pr. 2) Use image windows of size (2W+1)x(2W+1). 3) Select the pixel pr that maximizes a correlation function. HAVE TO SPECIFY: Region R, size W, and correlation function ψ.

Correlation-Based Methods plpl R(p l ) Left ImageRight Image For each pixel pl=[i,j] in the left image For each displacement d=[d1,d2] in R(pl) Compute The disparity of pl is the d that maximizes c(d) HAVE TO SPECIFY: Region R, size W, and correlation function ψ.

Correlation-Based Methods plpl R(p l ) Left ImageRight Image SUM OF SQUARED DIFFERENCES SSD CROSS-CORRELATION SSD is usually preferred: handles different intensity scales. Normalized cross-correlation is better (but is more expensive).

CSc D Computer Vision / Ioannis Stamos Correspondence Is Difficult Intensities in window may differ. Normalized cross-correlation may help.

CSc D Computer Vision / Ioannis Stamos Image Normalization  Even when the cameras are identical models, there can be differences in gain and sensitivity.  The cameras do not see exactly the same surfaces, so their overall light levels can differ.  For these reasons and more, it is a good idea to normalize the pixels in each window: From Sebastian Thrun/Jana Kosecka

CSc D Computer Vision / Ioannis StamosCorrespondence ( - ) 2 = ssd  

CSc D Computer Vision / Ioannis Stamos It is closely related to the SSD: Maximize Cross correlation Minimize Sum of Squared Differences Comparing Windows: From Jana Kosecka

CSc D Computer Vision / Ioannis Stamos Sum of squared differences Normalize cross-correlation Sum of absolute differences Region based Similarity Metrics Region based Similarity Metrics From Jana Kosecka

CSc D Computer Vision / Ioannis Stamos NCC score for two widely separated views NCC score From Jana Kosecka

CSc D Computer Vision / Ioannis Stamos Window size W = 3W = 20 Better results with adaptive window T. Kanade and M. Okutomi, A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment,, Proc. International Conference on Robotics and Automation, 1991.A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffusion. International Journal of Computer Vision, 28(2): , July 1998Stereo matching with nonlinear diffusion  Effect of window size (S. Seitz)

CSc D Computer Vision / Ioannis Stamos Stereo results Ground truthScene  Data from University of Tsukuba (Seitz)

CSc D Computer Vision / Ioannis Stamos Results with window correlation Window-based matching (best window size) Ground truth (Seitz)

CSc D Computer Vision / Ioannis Stamos Results with better method Boykov et al., Fast Approximate Energy Minimization via Graph Cuts,Fast Approximate Energy Minimization via Graph Cuts International Conference on Computer Vision, September (Seitz) Ground truth State of the art

CSc D Computer Vision / Ioannis Stamos Feature-Based Methods Left ImageRight Image Match sparse sets of extracted features. A feature descriptor for a line could contain: length l, orientation o, midpoint (x,y), average contrast c An example similarity measure (w’s are weights):

CSc D Computer Vision / Ioannis Stamos Correspondence Using Correlation LeftDisparity Map Images courtesy of Point Grey Research

CSc D Computer Vision / Ioannis Stamos LEFT IMAGE corner line structure Correspondence By Features From Sebastian Thrun/Jana Kosecka

Correspondence By Features RIGHT IMAGE corner line structure  Search in the right image… the disparity (dx, dy) is the displacement when the similarity measure is maximum From Sebastian Thrun/Jana Kosecka

CSc D Computer Vision / Ioannis Stamos Comparison of Matching Methods  Dense depth maps.  Need textured images  Sensitive to foreshorening/illumination changes  Need close views  Sparse depth maps.  Insensitive to illumination changes.  A-priori info used.  Faster. Problems: occlusions/spurious matches: =>Introduce constraints in matching (i.e. left-right consistency constraint) Correlation-BasedFeature-Based

CSc D Computer Vision / Ioannis Stamos Epipolar Constraint (Geometry) Center of projection Image plane πl Scene point Center of projection Epipoles OlOr P PlPr pl pr EPIPOLAR PLANE eler Image plane πr EPIPOLAR LINE

Epipolar Constraint Center of projection Image plane πl Scene point Center of projection Epipoles OlOr P PlPr pl pr EPIPOLAR PLANE eler Image plane πr EPIPOLAR LINE Extrinsic parameters: Left/Right Camera Frames: Pr=R(Pl-T), T=Or-Ol (1)

Epipolar Constraint Center of projection Image plane πl Scene point Center of projection Epipoles OlOr P PlPr pl pr EPIPOLAR PLANE eler Image plane πr EPIPOLAR LINE Given pl, pr is constrained to lie on the Epipolar Line (E.L.). For each left pixel pl, find the corresponding right E.L. Searching for pr reduces to a 1-D problem. Ol, Or, pl => Enough to define right E.L.

Epipolar Constraint Center of projection Image plane Scene points Center of projection Epipoles All E.L.s go through epipoles. Parallel image planes => epipoles at infinity. eler

Essential Matrix Estimate the epipolar geometry: correspondence between points and E.L.s. (1) Link bw/ epipolar constraint and extrinsic parameters of stereo system.

CSc D Computer Vision / Ioannis Stamos Essential Matrix Epipolar lines are found by Essential matrix Rank 2 Perspective Projection erel Perspective: p l =[x l,y l,f l ] T, p r =[x r,y r,z r ] T p l = f l /Z l P l, p r =f r /Z r P r

Camera Models (linear versions) World Point (Xw, Yw,Zw) Measured Pixel (xim, yim) Elegant decomposition. No distortion! ? Homogeneous Coordinates

CSc D Computer Vision / Ioannis Stamos Fundamental Matrix Camera to pixel coordinates: Essential matrix equation becomes: Epipolar lines: F: pixel coordinates ! E: camera coordinates ! erel M l (M r ) matrix of intrinsic parameters for left (right) camera. Fundamental matrix

CSc D Computer Vision / Ioannis StamosConclusions  Encodes information on extrinsic parameters.  Has rank 2.  Its 2 non-zero singular values are equal.  Encodes information on both the extrinsic and intrinsic parameters.  Has rank 2. Essential Matrix Fundamental Matrix

CSc D Computer Vision / Ioannis Stamos Estimating the epipolar geometry eler eler

CSc D Computer Vision / Ioannis Stamos Estimating the epipolar geometry Problem: Find the fundamental matrix from a set of image correspondences eler eler

CSc D Computer Vision / Ioannis Stamos Estimating the epipolar geometry With the respect to the constraint: Rank(F) = 2. eler

The 8-point algorithm n>=8 correspondences v: the 9 elements of F. A: n x 9 measurement matrix. Solve using SVD (solution up to a scale factor). Enforce rank(F)=2 =>SVD on the computed F. Be careful: numerical instabilities.

CSc D Computer Vision / Ioannis Stamos Epipolar Lines – Example

Example Two views Point Feature Matching From Jana Kosecka

CSc D Computer Vision / Ioannis Stamos Example Epipolar Geometry Camera Pose and Sparse Structure Recovery From Jana Kosecka

CSc D Computer Vision / Ioannis Stamos Locating the Epipoles from E & F eler F => el, er in pixel coordinates. E => el, er in camera coordinates. Fact: All epipolar lines pass through epipoles. Accurate epipole localization: 1)Refining epipolar lines. 2)Checking for consistency. 3)Uncalibrated stereo.

CSc D Computer Vision / Ioannis Stamos Image rectification  Given general displacement how to warp the views  Such that epipolar lines are parallel to each other  How to warp it back to canonical configuration (more details later) (more details later) (Seitz)

CSc D Computer Vision / Ioannis Stamos Epipolar rectification Rectified Image Pair Corresponding epipolar lines are aligned with the scan-lines Search for dense correspondence is a 1D search

CSc D Computer Vision / Ioannis Stamos Epipolar rectification Rectified Image Pair

CSc D Computer Vision / Ioannis Stamos Rectification (Trucco, Ch. 7)  Rotate left camera so that epipole goes to infinity (known R, known epipoles)  Apply same rotation to right camera  Rotate right camera by R  Adjust scale in both camera reference frames

CSc D Computer Vision / Ioannis Stamos Rectification  Problem: Epipolar lines not parallel to scan lines p l p r P OlOl OrOr elel erer PlPl PrPr Epipolar Plane Epipolar Lines Epipoles From Sebastian Thrun/Jana Kosecka

CSc D Computer Vision / Ioannis Stamos Rectification  Problem: Epipolar lines not parallel to scan lines p l p r P OlOl OrOr PlPl PrPr Epipolar Plane Epipolar Lines Epipoles at infinity Rectified Images From Sebastian Thrun/Jana Kosecka

CSc D Computer Vision / Ioannis Stamos 3-D Reconstruction Reprinted from “Stereo by Intra- and Intet-Scanline Search,” by Y. Ohta and T. Kanade, IEEE Trans. on Pattern Analysis and Machine Intelligence, 7(2): (1985).  1985 IEEE.

CSc D Computer Vision / Ioannis Stamos 3-D Reconstruction  Intrinsic and extrinsic  Intrinsic only  No information  Unambiguous (triangulation)  Up to unknown scaling factor  Up to unknown projective transformation A Priori Knowledge 3-D Reconstruction from two views

CSc D Computer Vision / Ioannis Stamos Projective Reconstruction Euclidean reconstruction Projective reconstruction From Sebastian Thrun/Jana Kosecka

CSc D Computer Vision / Ioannis Stamos Euclidean vs Projective reconstruction  Euclidean reconstruction – true metric properties of objects lenghts (distances), angles, parallelism are preserved  Unchanged under rigid body transformations  => Euclidean Geometry – properties of rigid bodies under rigid body transformations, similarity transformation  Projective reconstruction – lengths, angles, parallelism are NOT preserved – we get distorted images of objects – their distorted 3D counterparts --> 3D projective reconstruction  => Projective Geometry From Sebastian Thrun/Jana Kosecka

CSc D Computer Vision / Ioannis Stamos Check this out!

CSc D Computer Vision / Ioannis Stamos How can We Improve Stereo? Space-time stereo scanner uses unstructured light to aid in correspondence Result: Dense 3D mesh (noisy) From Sebastian Thrun/Jana Kosecka

CSc D Computer Vision / Ioannis Stamos Active Stereo: Adding Texture to Scene By James Davis, Honda Research, By James Davis, Honda Research, Now UCSC Now UCSC From Sebastian Thrun/Jana Kosecka

CSc D Computer Vision / Ioannis Stamos rectified Active Stereo (Structured Light) From Sebastian Thrun/Jana Kosecka

CSc D Computer Vision / Ioannis Stamos Range Images (depth images, depth maps, surface profiles, 2.5-D images) Sensors that produce depth directly. Pixel of a range image is the distance between a known reference frame and a visible point in the scene. Representations: Cloud of Points (x,y,z) Rij form (spatial information is explicit)

CSc D Computer Vision / Ioannis Stamos Active Range Sensors  Project energy or control sensor’s parameters.  Laser, Radars (accurate)/Sonars(inaccurate).  Active Focusing/Defocusing.

CSc D Computer Vision / Ioannis Stamos Triangulation

Triangulation Xc Yc Zc Image Light Stripe System. Light Plane: AX+BY+CZ+D=0 (in camera frame) Image Point: x=f X/Z, y=f Y/Z (perspective) Triangulation: Z=-D f/(A x + B y + C f) Move light stripe or object.

CSc D Computer Vision / Ioannis Stamos Time of Flight