COMP322/S2000/L271 Stereo Imaging Ref.V.S.Nalwa, A Guided Tour of Computer Vision, Addison Wesley, (ISBN ) Slides are adapted from CS641 (Spring 2000), offered by Dr. C. K Tang, CS Dept., HKUST

COMP322/S2000/L272 Stereo Imaging - Introduction l Two images l of the same static scene, each taken from a different viewpoint l can be used for 3-D reconstruction l this process is called stereo

COMP322/S2000/L273 Left EyeRight Eye Left Image Right Image Perceived Object 3-D Reconstruction by Stereo

COMP322/S2000/L274 Stereo Imaging in a Nutshell Correspondence Triangulation Alignment

COMP322/S2000/L275 Stereo Imaging l Theoretical Basis –Triangulation –Epipolar Geometry –Alignment, i.e. rectification l Correspondence - Matching of points from two images

COMP322/S2000/L276 + Left Center of Projection Object Point Left image Right Center of Projection + Right image Triangulation Line of sight

COMP322/S2000/L277 Triangulation - the underlying principle Assumptions: l perspective projection - pin-hole camera l uniqueness constraint along the line of sight Triangulation can be performed if m Cameras are calibrated (you know how to do this) m Correspondence of pixels between images are known (this is the major difficulty!!)

COMP322/S2000/L278 + Left Center of Projection Object Point Left image Right Center of Projection + Right image Triangulation Correspondence

COMP322/S2000/L279 Correspondence l Point (pixel) correspondence l Ambiguous correspondence between points in the 2 images l There may not be a correspondence pixel in the other image. Why?

COMP322/S2000/L Right Image Right Center of Projection + Left Image Left Center of Projection Correspondence - Ambiguous points

COMP322/S2000/L2711 Left Center of Projection Right Center of Projection ++ Left ImageRight Image Occluded in Right Image Occluded in Left Image Occlusion - no corresponding points

COMP322/S2000/L2712 Correspondence l Brute force approach: O(n 2 ) search, where n is the image dimension, i.e., one pixel is compared with every pixel in the other image, and return the best match. l this O(n 2 ) search is actually unnecessary: l 2 images of a static scene are related by Epipolar Geometry l the search of corresponding pixels becomes O(n) by making use of epipolar geometry

COMP322/S2000/L Right Image Right Center of Projection Projection of l - epipolar line + l Left Image Left Center of Projection Epipolar Geometry

COMP322/S2000/L2714 Correspondence based on Epipolar Constraint l one point in an image (e.g., left image) corresponds to a line in the other image (e.g. right image), this line is called the epipolar line l hence, given a point in image (I 1 ), to find a corresponding matching point in the other image (I 2 ), we can l compute the epipolar line in I 2 (and its conjugate in I 1 ) l search along the epipolar line

COMP322/S2000/L Left Center of Projection O 1 + Right Center of Projection O 2 Left image Right image Epipolar Lines Baseline Epipolar Plane Object Point

COMP322/S2000/L2716 Epipolar Plane l The 2 centers of projections and the object point (or one image point) together define a plane l this plane is called Epipolar Plane l this plane intersects with the 2 images and gives the corresponding epipolar lines

COMP322/S2000/L2717 How Does Stereo Work Epipolar Geometry Triangulation Alignment

COMP322/S2000/L2718 Alignment - Rectification l In general, the image planes of a given stereo pair are not parallel l By the process of rectification, l the image planes become coplanar and parallel to the baseline l corresponding epipolar lines are horizontally coincident

COMP322/S2000/L2719 Example of Rectification After: Before:

COMP322/S2000/L2720 Rectification l So, the 1-D search can proceed along the same, corresponding scanline l Hence, the correspondence problem is translated into one of finding the disparity l Note: Still under the assumption that the cameras are calibrated.

COMP322/S2000/L2721 Rectification l the image planes become coplanar and parallel to the baseline l corresponding epipolar lines are horizontally coincident

COMP322/S2000/L2722 Rectification l2l2 r2r2 l1l1 r1r1 v1 v2 Each “pair” of epipolar line will be transformed to be aligned (or the image planes will be “rotated” to be coplanar and parallel to the base line

COMP322/S2000/L2723 Correspondence and Disparity If the stereo pair has been rectified, then, for each image point (u,v) on one image: –1-D search along the same corresponding scanline –and find the disparity D(u,v) u1u1 u2u2

COMP322/S2000/L2724 v u m1m1 v u + d m2m2 correlation windows Epipolar line Intensity-based method

COMP322/S2000/L2725 Minimize Error Function Block matching: find minimum squared error

COMP322/S2000/L2726 How Does Stereo Work Correspondence Triangulation Alignment - Rectification