Stereo Vision ECE 847: Digital Image Processing Stan Birchfield Clemson University

Modeling from multiple views time # cameras photograph binocular stereo trinocular stereo multi-baseline stereo camcorder human vision camera dome two frames...  – Greek for solid

Stereoscope Invented by Wheatstone in 1838

Modern version

Can you fuse these? rightleft No special instrument needed Just relax your eyes L R

Random dot stereogram invented by Bela Julesz in

Autostereogram Do you see the shark?

Can you cross-fuse these? rightleft Note: Cross-fusion is necessary if distance between images is greater than inter-ocular distance L R R L impossible: instead, trick the brain: Tsukuba stereo images courtesy of Y. Ohta and Y. Nakamura at the University of Tsukuba

Human stereo geometry fixation point corresponding points aRaR aLaL disparity

Horopter Horopter: surface where disparity is zero For round retina, the theoretical horopter is a circle (Vieth-Muller circle)

Cyclopean image

Panum’s fusional area (volume) Human visual system is only capable of fusing the two images with a narrow range of disparities around fixation point This area (volume) is Panum’s fusional area Outside this area we get double-vision (diplopia)

Human visual pathway

photos courtesy California Academy of Science Cheetah: More accurate depth estimation Antelope: larger field of view Prey and predator

Stereo geometry for pinhole cameras with flat retinas C,C’,x,x’ and X are coplanar Left cameraRight camera world point center of projection epipolar plane epipolar line for x epipole baseline M. Pollefeys,

epipoles e,e’ = intersection of baseline with image plane = projection of projection center in other image = vanishing point of camera motion direction an epipolar plane = plane containing baseline (1-D family) an epipolar line = intersection of epipolar plane with image (always come in corresponding pairs) Epipolar geometry M. Pollefeys,

What if only C,C’,x are known? Epipolar geometry epipole center of projection baseline All points on p project on l and l’ M. Pollefeys,

Family of planes  and lines l and l’ Intersection in e and e’ Epipolar geometry M. Pollefeys,

Example: Converging cameras M. Pollefeys,

e e’ Example: Forward motion M. Pollefeys,

Epipolar geometry and Fundamental matrix epipolar line (epipole: intersection of all epipolar lines) (computed with 20 points)

Epipolar geometry and Fundamental matrix epipolar line (epipole: intersection of all epipolar lines) (computed with 50 points)

Fundamental matrix point in image 1 point in image 2 fundamental matrix

Epipolar lines (1) epipolar line in image 2 associated with (x,y) in image 1

Epipolar lines (2) epipolar line in image 1 associated with (x’,y’) in image 2

Computing the fundamental matrix known unknown

Computing the fundamental matrix known unknown

Computing the fundamental matrix 1.Construct A (nx9) from correspondences 2.Compute SVD of A: A = U  V T 3.Unit-norm solution of Af=0 is given by v n (the right-most singular vector of A) 4.Reshape f into F 1 5.Compute SVD of F 1 : F 1 =U F  F V F T 6.Set  F (3,3)=0 to get  F ’ (The enforces rank(F)=2) 7.Reconstruct F=U F  F ’V F T 8.Now x T F is epipolar line in image 2, and Fx’ is epipolar line in image 1

(simple for stereo  rectification) Example: Motion parallel to image plane M. Pollefeys,

Example: Motion parallel to image scanlines Epipoles are at infinity Scanlines are the epipolar lines In this case, the images are said to be “rectified” Tsukuba stereo images courtesy of Y. Ohta and Y. Nakamura at the University of Tsukuba

Standard (rectified) stereo geometry pure translation along X-axis M. Pollefeys,

Perspective projection X x X x M. Pollefeys,

Rectified geometry xLxL xRxR XLXL -X R b

Standard stereo geometry disparity is inversely proportional to depth stereo vision is less useful for distant objects M. Pollefeys,

Binocular rectified stereo rightleft disparity mapdepth discontinuities epipolar constraint

Matching scanlines intensity L R disparity lamp wall pixel rightleft

Stereo matching Search is limited to epipolar line (1D) Look for most similar pixel ? M. Pollefeys,

Aggregation Use more than one pixel Assume neighbors have similar disparities * –Use correlation window containing pixel –Allows to use SSD, ZNCC, Census, etc. M. Pollefeys,

Block matching

Dissimilarity measures Connection between SSD and cross correlation: Also normalized correlation, rank, census, sampling-insensitive... Most common:

More efficient implementation Key idea: Summation over window is correlation with box filter, which is separable Running sum improves efficiency even more Note: w is half-width

Compare intensities pixel-by-pixel Comparing image regions I(x,y) I´(x,y) Sum of Square Differences Dissimilarity measures Note: SAD is fast approximation (replace square with absolute value) M. Pollefeys,

Compare intensities pixel-by-pixel Comparing image regions I(x,y) I´(x,y) Dissimilarity measures If energy does not change much, then minimizing SSD equals maximizing cross-correlation: M. Pollefeys,

Compare intensities pixel-by-pixel Comparing image regions I(x,y) I´(x,y) Zero-mean Normalized Cross Correlation Similarity measures M. Pollefeys,

Compare intensities pixel-by-pixel Comparing image regions I(x,y) I´(x,y) Census Similarity measures (Real-time chip from TYZX based on Census) only compare bit signature M. Pollefeys,

Sampling-Insensitive Pixel Dissimilarity d(x L,x R ) xLxL xRxR d(x L,x R ) = min{d(x L,x R ),d(x R,x L )}Our dissimilarity measure: [Birchfield & Tomasi 1998] ILIL IRIR

Given: An interval A such that [x L – ½, x L + ½] _ A, and [x R – ½, x R + ½] _ A Dissimilarity Measure Theorems If | x L – x R | ≤ ½, then d(x L,x R ) = 0 | x L – x R | ≤ ½ iff d(x L,x R ) = 0 ∩ ∩ Theorem 1: Theorem 2: (when A is convex or concave) (when A is linear)

Aggregation window sizes Small windows disparities similar more ambiguities accurate when correct Large windows larger disp. variation more discriminant often more robust use shiftable windows to deal with discontinuities (Illustration from Pascal Fua)

Occlusions (Slide from Pascal Fua)

Left-right consistency check Search left-to-right, then right-to-left Retain disparity only if they agree xLxL d Do minima coincide?

Results: correlation disparity mapleft with left-right consistency check

Constraints Epipolar – match must lie on epipolar line Piecewise constancy – neighboring pixels should usually have same disparity Piecewise continuity – neighboring pixels should usually have similar disparity Disparity – impose allowable range of disparities (Panum’s fusional area) Disparity gradient – restricts slope of disparity Figural continuity – disparity of edges across scanlines Uniqueness – each pixel has no more than one match (violated by windows and mirrors) Ordering – disparity function is monotonic (precludes thin poles)

Exploiting scene constraints M. Pollefeys,

Ordering constraint , , , , surface slice surface as a path occlusion right occlusion left M. Pollefeys,

Uniqueness constraint In an image pair each pixel has at most one corresponding pixel –In general one corresponding pixel –In case of occlusion there is none M. Pollefeys,

Disparity constraint surface slice surface as a path bounding box disparity band use reconstructed features to determine bounding box constant disparity surfaces M. Pollefeys,

Figural continuity constraint rightleft [University of Tsukuba]

Cooperative algorithm

Disparity space image

Dynamic Programming: 1D Search D isparity map occlusion depth discontinuity RIGHT LEFT cart c a t string editing: stereo matching: penalties: mismatch = 1 insertion = 1 deletion = 1 c a t c a r t

Minimizing a 2D Cost Functional Minimize: disparity 2D: GLOBAL disparity pixel ? 1D: Global u(l ) p,q Discontinuity penalty: l p,q minimum cut = disparity surface solves LOCAL Local (GOOD) (BAD) 

Multiway-Cut: 2D Search pixels labels pixels labels [Boykov, Veksler, Zabih 1998]

Multiway-Cut Algorithm minimum cut Minimizes source label sink label pixels (cost of label discontinuity) (cost of assigning label to pixel) pixels labels

Energy minimization (Slide from Pascal Fua)

Graph Cut (Slide from Pascal Fua) (general formulation requires multi-way cut!)

(Boykov et al ICCV‘99) (Roy and Cox ICCV‘98) Simplified graph cut

Correspondence as Segmentation Problem: disparities (fronto-parallel)O(  ) surfaces (slanted) O(   2 n) => computationally intractable! Solution: iteratively determine which labels to use label pixels find affine parameters of regions multiway-cut (Expectation) Newton-Raphson (Maximization)

Stereo Results (Dynamic Programming)

Stereo Results (Multiway-Cut)

Stereo Results on Middlebury Database image Birchfield Tomasi 1999 Hong- Chen 2004

Untextured regions remain a challenge Multiway-cutDynamic programming

Results: dynamic programming disparity map [Bobick & Intille] left

Results: multiway cut disparity map left [Kolmogorov & Zabih]

Results: multiway cut (untextured) disparity map

Multi-camera configurations Okutami and Kanade (illustration from Pascal Fua) M. Pollefeys,

Example: Tsukuba Tsukuba dataset

Real-time stereo on GPU Computes Sum-of-Square-Differences (use pixelshader) Hardware mip-map generation for aggregation over window Trade-off between small and large support window (Yang and Pollefeys, CVPR2003) 290M disparity hypothesis/sec (Radeon9800pro) e.g. 512x512x36disparities at 30Hz GPU is great for vision too!

Stereo matching Optimal path (dynamic programming ) Similarity measure (SSD or NCC) Constraints epipolar ordering uniqueness disparity limit Trade-off Matching cost (data) Discontinuities (prior) Consider all paths that satisfy the constraints pick best using dynamic programming M. Pollefeys,

Hierarchical stereo matching Downsampling (Gaussian pyramid) Disparity propagation Allows faster computation Deals with large disparity ranges M. Pollefeys,

Disparity map image I(x,y) image I´(x´,y´) Disparity map D(x,y) (x´,y´)=(x+D(x,y),y) M. Pollefeys,

Example: reconstruct image from neighboring images M. Pollefeys,

Stereo matching with general camera configuration M. Pollefeys,

Image pair rectification M. Pollefeys,

Planar rectification Bring two views to standard stereo setup (moves epipole to  ) (not possible when in/close to image) ~ image size (calibrated) Distortion minimization (uncalibrated) M. Pollefeys,

polar rectification planar rectification original image pair M. Pollefeys,

Stereo camera configurations (Slide from Pascal Fua)

More cameras Multi-baseline stereo [Okutomi & Kanade]