Computer Vision Multiple View Geometry & Stereo Marc Pollefeys COMP 256

Computer Vision C1C1 C2C2 l2l2  l1l1 e1e1 e2e2 Fundamental matrix (3x3 rank 2 matrix) 1.Computable from corresponding points 2.Simplifies matching 3.Allows to detect wrong matches 4.Related to calibration Underlying structure in set of matches for rigid scenes l2l2 C1C1 p1p1 L1L1 p2p2 L2L2 M C2C2 p1p1 p2p2 C1C1 C2C2 l2l2  l1l1 e1e1 e2e2 p1p1 L1L1 p2p2 L2L2 M l2l2 lT1lT1 Canonical representation: Last class: epipolar geometry

Computer Vision Aug 26/28-Introduction Sep 2/4CamerasRadiometry Sep 9/11Sources & ShadowsColor Sep 16/18Linear filters & edges(Isabel hurricane) Sep 23/25Pyramids & TextureMulti-View Geometry Sep30/Oct2StereoProject proposals Oct 7/9-Optical flow Oct 14/16Tracking- Oct 21/23Silhouettes/carvingStructure from motion Oct 28/30-Camera calibration Nov 4/6Project updateSegmentation Nov 11/13FittingProbabilistic segm.&fit. Nov 18/20Matching templatesMatching relations Nov 25/27Range data(Thanksgiving) Dec 2/4Final project Tentative class schedule

Computer Vision Multiple Views (Faugeras and Mourrain, 1995)

Computer Vision Two Views Epipolar Constraint

Computer Vision Three Views Trifocal Constraint

Computer Vision Four Views Quadrifocal Constraint (Triggs, 1995)

Computer Vision Geometrically, the four rays must intersect in P..

Computer Vision Quadrifocal Tensor and Lines

Computer Vision Quadrifocal tensor determinant is multilinear thus linear in coefficients of lines ! There must exist a tensor with 81 coefficients containing all possible combination of x,y,w coefficients for all 4 images: the quadrifocal tensor

Computer Vision STEREOPSIS Reading: Chapter 11. The Stereopsis Problem: Fusion and Reconstruction Human Stereopsis and Random Dot Stereograms Cooperative Algorithms Correlation-Based Fusion Multi-Scale Edge Matching Dynamic Programming Using Three or More Cameras

Computer Vision An Application: Mobile Robot Navigation The Stanford Cart, H. Moravec, The INRIA Mobile Robot, Courtesy O. Faugeras and H. Moravec.

Computer Vision Reconstruction / Triangulation

Computer Vision (Binocular) Fusion

Computer Vision Reconstruction Linear Method: find P such that Non-Linear Method: find Q minimizing

Computer Vision Rectification All epipolar lines are parallel in the rectified image plane.

Computer Vision Image pair rectification simplify stereo matching by warping the images Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole e to (1,0,0) try to minimize image distortion problem when epipole in (or close to) the image

Computer Vision Planar rectification Bring two views to standard stereo setup (moves epipole to  ) (not possible when in/close to image) ~ image size (calibrated) Distortion minimization (uncalibrated) (standard approach)

Computer Vision

Computer Vision

Computer Vision Polar re-parameterization around epipoles Requires only (oriented) epipolar geometry Preserve length of epipolar lines Choose  so that no pixels are compressed original image rectified image Polar rectification (Pollefeys et al. ICCV’99) Works for all relative motions Guarantees minimal image size

Computer Vision polar rectification: example

Computer Vision polar rectification: example

Computer Vision Example: Béguinage of Leuven Does not work with standard Homography-based approaches

Computer Vision Example: Béguinage of Leuven

Computer Vision Reconstruction from Rectified Images Disparity: d=u’-u.Depth: z = -B/d.

Computer Vision Stereopsis Figure from US Navy Manual of Basic Optics and Optical Instruments, prepared by Bureau of Naval Personnel. Reprinted by Dover Publications, Inc., 1969.

Computer Vision Human Stereopsis: Reconstruction Disparity: d = r-l = D-F. d=0 d<0 In 3D, the horopter.

Computer Vision Human Stereopsis: experimental horopter…

Computer Vision Iso-disparity curves: planar retinas X0X0 X1X1 C1C1 X∞X∞ C2C2 XiXi XjXj the retina act as if it were flat!

Computer Vision What if F is not known? Human Stereopsis: Reconstruction Helmoltz (1909): There is evidence showing the vergence angles cannot be measured precisely. Humans get fooled by bas-relief sculptures. There is an analytical explanation for this. Relative depth can be judged accurately.

Computer Vision Human Stereopsis: Binocular Fusion How are the correspondences established? Julesz (1971): Is the mechanism for binocular fusion a monocular process or a binocular one?? There is anecdotal evidence for the latter (camouflage). Random dot stereograms provide an objective answer BP!

Computer Vision A Cooperative Model (Marr and Poggio, 1976) Excitory connections: continuity Inhibitory connections: uniqueness Iterate: C =  C - w  C + C. ei0 Reprinted from Vision: A Computational Investigation into the Human Representation and Processing of Visual Information by David Marr.  1982 by David Marr. Reprinted by permission of Henry Holt and Company, LLC.

Computer Vision Correlation Methods (1970--) Normalized Correlation: minimize  instead. Slide the window along the epipolar line until w.w’ is maximized. 2 Minimize |w-w’|.

Computer Vision Correlation Methods: Foreshortening Problems Solution: add a second pass using disparity estimates to warp the correlation windows, e.g. Devernay and Faugeras (1994). Reprinted from “Computing Differential Properties of 3D Shapes from Stereopsis without 3D Models,” by F. Devernay and O. Faugeras, Proc. IEEE Conf. on Computer Vision and Pattern Recognition (1994).  1994 IEEE.

Computer Vision Multi-Scale Edge Matching (Marr, Poggio and Grimson, ) Edges are found by repeatedly smoothing the image and detecting the zero crossings of the second derivative (Laplacian). Matches at coarse scales are used to offset the search for matches at fine scales (equivalent to eye movements).

Computer Vision Multi-Scale Edge Matching (Marr, Poggio and Grimson, ) One of the two input images Image Laplacian Zeros of the Laplacian Reprinted from Vision: A Computational Investigation into the Human Representation and Processing of Visual Information by David Marr.  1982 by David Marr. Reprinted by permission of Henry Holt and Company, LLC.

Computer Vision Multi-Scale Edge Matching (Marr, Poggio and Grimson, ) Reprinted from Vision: A Computational Investigation into the Human Representation and Processing of Visual Information by David Marr.  1982 by David Marr. Reprinted by permission of Henry Holt and Company, LLC.

Computer Vision The Ordering Constraint But it is not always the case.. In general the points are in the same order on both epipolar lines.

Computer Vision Dynamic Programming (Baker and Binford, 1981) Find the minimum-cost path going monotonically down and right from the top-left corner of the graph to its bottom-right corner. Nodes = matched feature points (e.g., edge points). Arcs = matched intervals along the epipolar lines. Arc cost = discrepancy between intervals.

Computer Vision Dynamic Programming (Ohta and Kanade, 1985) Reprinted from “Stereo by Intra- and Intet-Scanline Search,” by Y. Ohta and T. Kanade, IEEE Trans. on Pattern Analysis and Machine Intelligence, 7(2): (1985).  1985 IEEE.

Computer Vision Three Views The third eye can be used for verification..

Computer Vision More Views (Okutami and Kanade, 1993) Pick a reference image, and slide the corresponding window along the corresponding epipolar lines of all other images, using inverse depth relative to the first image as the search parameter. Use the sum of correlation scores to rank matches. Reprinted from “A Multiple-Baseline Stereo System,” by M. Okutami and T. Kanade, IEEE Trans. on Pattern Analysis and Machine Intelligence, 15(4): (1993). \copyright 1993 IEEE.

Computer Vision Stereo matching Optimal path (dynamic programming ) Similarity measure (SSD or NCC) Constraints epipolar ordering uniqueness disparity limit disparity gradient limit Trade-off Matching cost (data) Discontinuities (prior) (Cox et al. CVGIP’96; Koch’96; Falkenhagen´97; Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)

Computer Vision Hierarchical stereo matching Downsampling (Gaussian pyramid) Disparity propagation Allows faster computation Deals with large disparity ranges ( Falkenhagen´97;Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)

Computer Vision Disparity map image I(x,y) image I´(x´,y´) Disparity map D(x,y) (x´,y´)=(x+D(x,y),y)

Computer Vision Example: reconstruct image from neighboring images

Computer Vision I1I2I10 Reprinted from “A Multiple-Baseline Stereo System,” by M. Okutami and T. Kanade, IEEE Trans. on Pattern Analysis and Machine Intelligence, 15(4): (1993). \copyright 1993 IEEE.

Computer Vision Multi-view depth fusion Compute depth for every pixel of reference image –Triangulation –Use multiple views –Up- and down sequence –Use Kalman filter (Koch, Pollefeys and Van Gool. ECCV‘98) Allows to compute robust texture

Computer Vision Real-time stereo on graphics hardware Computes Sum-of-Square-Differences Hardware mip-map generation used to aggregate results over support region Trade-off between small and large support window Ruigang Yang and Marc Pollefeys 140M disparity hypothesis/sec on Radeon 9700pro e.g. 512x512x20disparities at 30Hz Shape of a kernel for summing up 6 levels

Computer Vision Sample Re-Projections near far

Computer Vision (1x1) (1x1+2x2) (1x1+2x2 +4x4+8x8) (1x1+2x2 +4x4+8x8 +16x16) Combine multiple aggregation windows using hardware mipmap and multiple texture units in single pass

Computer Vision Live stereo demo PC with ATI Radeon 9700pro Bumblebee (Point Grey) OpenGL 1.4 (Fragment programs) Code available at:

Computer Vision Cool ideas Space-time stereo (varying illumination, not shape)

Computer Vision More on stereo … The Middleburry Stereo Vision Research Page Recommended reading D. Scharstein and R. Szeliski. A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms. IJCV 47(1/2/3):7-42, April-June PDF file (1.15 MB) - includes current evaluation. Microsoft Research Technical Report MSR-TR , November PDF file (1.27 MB).PDF file

Computer Vision Next class: project proposals!