EECS 274 Computer Vision Stereopsis

Stereopsis Stereopsis: Fusion and Reconstruction Correlation-Based Fusion Multi-Scale Edge Matching Dynamic Programming Using Three or More Cameras Reading: FP Chapter 11, S Chapter 11

An application: mobile robot navigation The INRIA Mobile Robot, 1990. The Stanford Cart, H. Moravec, 1979. Courtesy O. Faugeras and H. Moravec.

Stereo vision Fusion: match points observed by two or more cameras Reconstruction: find the pre-image of the matching points in 3D world Assume calibrated camera and essential matrix/trifocal tensors associated with 3 cameras are known

Reconstruction / triangulation

(Binocular) Fusion

Non-Linear Method: find Q minimizing Reconstruction Geometric approach Algebraic approach Linear Method: find P such that Non-Linear Method: find Q minimizing

Rectification All epipolar lines are parallel in the rectified image plane

Reconstruction from rectified images Disparity: d=u’-u Depth: z = -B/d × f

Correlation methods (1970--) Slide the window along the epipolar line until w.w’ is maximized. 2 Minimize |w-w’| Normalized Correlation: minimize q instead.

Correlation methods: foreshortening problems Solution: add a second pass using disparity estimates to warp the correlation windows, e.g. Devernay and Faugeras (1994)

Multi-scale edge matching (Marr, Poggio and Grimson, 1979-81) Edges are found by repeatedly smoothing the image and detecting the zero crossings of the second derivative (Laplacian) Matches at coarse scales are used to offset the search for matches at fine scales (equivalent to eye movements)

Multi-scale edge matching (Marr, Poggio and Grimson, 1979-81) One of the two input images Image Laplacian Zeros of the Laplacian

Multi-scale edge matching (Marr, Poggio and Grimson, 1979-81)

The ordering constraint In general the points are in the same order on both epipolar lines. But it is not always the case..

Ordering constraints points are not necessarily in order d-b-a c’-b’-d’

Dynamic programming (Baker and Binford, 1981) Find the minimum-cost path going monotonically down and right from the top-left corner of the graph to its bottom-right corner. Nodes = matched feature points (e.g., edge points). Arcs = matched intervals along the epipolar lines. Arc cost = discrepancy between intervals.

Dynamic programming (Ohta and Kanade, 1985) use inter-scanline f or point correspondence on vertical edges use DP for both intra-scanline and inter-scanline search

The third eye can be used for verification.. Three views The third eye can be used for verification.. b1-a2 match is wrong as thee is no corresponding point in camera 3

More views (Okutami and Kanade, 1993) Pick a reference image, and slide the corresponding window along the corresponding epipolar lines of all other images, using inverse depth relative to the first image as the search parameter. Use the sum of correlation scores to rank matches.

I1 I2 I10

Correspondence Extensive literature Smooth surfaces Region based Graph cut Mutual information Belief propagation Conditional random field Smooth surfaces

Middlebury stereo dataset De facto data set with ground truth, code, and comprehensive performance evaluation http://vision.middlebury.edu/stereo/data/

Performance evaluation