Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset

Geometric Computer Vision course schedule (tentative) LectureExercise Sept 16Introduction- Sept 23Geometry & Camera modelCamera calibration Sept 30Single View Metrology (Changchang Wu) Measuring in images Oct. 7Feature Tracking/MatchingCorrespondence computation Oct. 14Epipolar GeometryF-matrix computation Oct. 21Shape-from-SilhouettesVisual-hull computation Oct. 28Stereo matchingpapers Nov. 4Structure from motion and visual SLAMProject proposals Nov. 11Multi-view geometry and self-calibration Papers Nov. 18Shape-from-XPapers Nov. 25Structured light and active range sensing Papers Dec. 23D modeling, registration and range/depth fusion (Christopher Zach?) Papers Dec. 9Appearance modeling and image- based rendering Papers Dec. 16Final project presentations

Stereo Standard stereo geometry Stereo matching Aggregation Optimization (1D, 2D) General camera configuration Rectifications Plane-sweep Multi-view stereo

Real-time stereo on GPU Computes Sum-of-Square-Differences (use pixelshader) Hardware mip-map generation for aggregation over window Trade-off between small and large support window (Yang and Pollefeys, CVPR2003) 290M disparity hypothesis/sec (Radeon9800pro) e.g. 512x512x36disparities at 30Hz GPU is great for vision too!

Stereo

6 Challenges Ill-posed inverse problem Recover 3-D structure from 2-D information Difficulties Uniform regions Half-occluded pixels

7 Pixel Dissimilarity Absolute difference of intensities c=|I1(x,y)- I2(x-d,y)| Interval matching [Birchfield 98] Considers sensor integration Represents pixels as intervals

8 Alternative Dissimilarity Measures Rank and Census transforms [Zabih ECCV94] Rank transform: Define window containing R pixels around each pixel Count the number of pixels with lower intensities than center pixel in the window Replace intensity with rank (0..R-1) Compute SAD on rank-transformed images Census transform: Use bit string, defined by neighbors, instead of scalar rank Robust against illumination changes

9 Rank and Census Transform Results Noise free, random dot stereograms Different gain and bias

10 Systematic Errors of Area-based Stereo Ambiguous matches in textureless regions Surface over-extension [Okutomi IJCV02]

11 Surface Over-extension Expected value of E[(x-y) 2 ] for x in left and y in right image is: Case A: σ F 2 + σ B 2 +(μ F - μ B ) 2 for w/2-λ pixels in each row Case B: 2 σ B 2 for w/2+λ pixels in each row Right image Left image Disparity of back surface

12 Surface Over-extension Discontinuity perpendicular to epipolar lines Discontinuity parallel to epipolar lines Right image Left image Disparity of back surface

13 Over-extension and shrinkage Turns out that: for discontinuities perpendicular to epipolar lines And: for discontinuities parallel to epipolar lines

14 Random Dot Stereogram Experiments

15 Random Dot Stereogram Experiments

16 Offset Windows  Equivalent to using min nearby cost  Result: loss of depth accuracy

17 Discontinuity Detection Use offset windows only where appropriate Bi-modal distribution of SSD Pixel of interest different than mode within window

18 Compact Windows [Veksler CVPR03]: Adapt windows size based on: Average matching error per pixel Variance of matching error Window size (to bias towards larger windows) Pick window that minimizes cost

19 Integral Image Sum of shaded part Compute an integral image for pixel dissimilarity at each possible disparity AC DB Shaded area = A+D-B-C Independent of size

20 Results using Compact Windows

21 Rod-shaped filters Instead of square windows aggregate cost in rod-shaped shiftable windows [Kim CVPR05] Search for one that minimizes the cost (assume that it is an iso-disparity curve) Typically use 36 orientations

22 Locally Adaptive Support Apply weights to contributions of neighboring pixels according to similarity and proximity [Yoon CVPR05]

23 Locally Adaptive Support Similarity in CIE Lab color space: Proximity: Euclidean distance Weights:

24 Locally Adaptive Support: Results Locally Adaptive Support

25 Locally Adaptive Support: Results

Occlusions (Slide from Pascal Fua)

Exploiting scene constraints

Ordering constraint , , , , surface slice surface as a path occlusion right occlusion left

Uniqueness constraint In an image pair each pixel has at most one corresponding pixel In general one corresponding pixel In case of occlusion there is none

Disparity constraint surface slice surface as a path bounding box disparity band use reconstructed features to determine bounding box constant disparity surfaces

Stereo matching Optimal path (dynamic programming ) Similarity measure (SSD or NCC) Constraints epipolar ordering uniqueness disparity limit Trade-off Matching cost (data) Discontinuities (prior) Consider all paths that satisfy the constraints pick best using dynamic programming

Hierarchical stereo matching Downsampling (Gaussian pyramid) Disparity propagation Allows faster computation Deals with large disparity ranges

Disparity map image I(x,y) image I´(x´,y´) Disparity map D(x,y) (x´,y´)=(x+D(x,y),y)

Example: reconstruct image from neighboring images

36 Semi-global optimization Optimize: E=E data +E(|D p -D q |=1)+E(|D p -D q |>1) [Hirshmüller CVPR05] Use mutual information as cost NP-hard using graph cuts or belief propagation (2-D optimization) Instead do dynamic programming along many directions Don’t use visibility or ordering constraints Enforce uniqueness Add costs

37 Results of Semi-global optimization

38 Results of Semi-global optimization No. 1 overall in Middlebury evaluation (at 0.5 error threshold as of Sep. 2006)

Energy minimization (Slide from Pascal Fua)

Graph Cut (Slide from Pascal Fua) (general formulation requires multi-way cut!)

(Boykov et al ICCV‘99) (Roy and Cox ICCV‘98) Simplified graph cut

Belief Propagation per pixel +left +right +up +down subsequent iterations (adapted from J. Coughlan slides) first iteration Belief of one node about another gets propagated through messages (full pdf, not just most likely state)

Stereo matching with general camera configuration

Image pair rectification

Planar rectification Bring two views to standard stereo setup (moves epipole to  ) (not possible when in/close to image) ~ image size (calibrated) Distortion minimization (uncalibrated)

Polar re-parameterization around epipoles Requires only (oriented) epipolar geometry Preserve length of epipolar lines Choose  so that no pixels are compressed original image rectified image Polar rectification (Pollefeys et al. ICCV’99) Works for all relative motions Guarantees minimal image size

polar rectification planar rectification original image pair

Example: Béguinage of Leuven Does not work with standard Homography-based approaches

Example: Béguinage of Leuven

General iso-disparity surfaces ( Pollefeys and Sinha, ECCV’04) Example: polar rectification preserves disp. Application: Active vision Also interesting relation to human horopter

Stereo camera configurations (Slide from Pascal Fua)

Multi-camera configurations Okutami and Kanade (illustration from Pascal Fua)

Multi-baseline, multi-resolution At each depth, baseline and resolution selected proportional to that depth Allows to keep depth accuracy constant! Variable Baseline/Resolution Stereo (Gallup et al., CVPR08)

Variable Baseline/Resolution Stereo: comparison

Multi-view depth fusion Compute depth for every pixel of reference image Triangulation Use multiple views Up- and down sequence Use Kalman filter (Koch, Pollefeys and Van Gool. ECCV‘98) Allows to compute robust texture

Plane-sweep multi-view matching Simple algorithm for multiple cameras no rectification necessary doesn’t deal with occlusions Collins’96; Roy and Cox’98 (GC); Yang et al.’02/’03 (GPU)

Next class: structured light and active scanning