1. Stereo Class 7 Read Chapter 7 of tutorial http://cat.middlebury.edu/stereo/ Tsukuba dataset

2. Geometric Computer Vision course schedule (tentative) LectureExercise Sept 16Introduction- Sept 23Geometry & Camera modelCamera calibration Sept 30Single View Metrology (Changchang Wu) Measuring in images Oct. 7Feature Tracking/MatchingCorrespondence computation Oct. 14Epipolar GeometryF-matrix computation Oct. 21Shape-from-SilhouettesVisual-hull computation Oct. 28Stereo matchingpapers Nov. 4Structure from motion and visual SLAMProject proposals Nov. 11Multi-view geometry and self-calibration Papers Nov. 18Shape-from-XPapers Nov. 25Structured light and active range sensing Papers Dec. 23D modeling, registration and range/depth fusion (Christopher Zach?) Papers Dec. 9Appearance modeling and image- based rendering Papers Dec. 16Final project presentations

3. Stereo Standard stereo geometry Stereo matching Aggregation Optimization (1D, 2D) General camera configuration Rectifications Plane-sweep Multi-view stereo

4. Real-time stereo on GPU Computes Sum-of-Square-Differences (use pixelshader) Hardware mip-map generation for aggregation over window Trade-off between small and large support window (Yang and Pollefeys, CVPR2003) 290M disparity hypothesis/sec (Radeon9800pro) e.g. 512x512x36disparities at 30Hz GPU is great for vision too!

5. Stereo

6. 6 Challenges Ill-posed inverse problem Recover 3-D structure from 2-D information Difficulties Uniform regions Half-occluded pixels

7. 7 Pixel Dissimilarity Absolute difference of intensities c=|I1(x,y)- I2(x-d,y)| Interval matching [Birchfield 98] Considers sensor integration Represents pixels as intervals

8. 8 Alternative Dissimilarity Measures Rank and Census transforms [Zabih ECCV94] Rank transform: Define window containing R pixels around each pixel Count the number of pixels with lower intensities than center pixel in the window Replace intensity with rank (0..R-1) Compute SAD on rank-transformed images Census transform: Use bit string, defined by neighbors, instead of scalar rank Robust against illumination changes

9. 9 Rank and Census Transform Results Noise free, random dot stereograms Different gain and bias

10. 10 Systematic Errors of Area-based Stereo Ambiguous matches in textureless regions Surface over-extension [Okutomi IJCV02]

11. 11 Surface Over-extension Expected value of E[(x-y) 2 ] for x in left and y in right image is: Case A: σ F 2 + σ B 2 +(μ F - μ B ) 2 for w/2-λ pixels in each row Case B: 2 σ B 2 for w/2+λ pixels in each row Right image Left image Disparity of back surface

12. 12 Surface Over-extension Discontinuity perpendicular to epipolar lines Discontinuity parallel to epipolar lines Right image Left image Disparity of back surface

13. 13 Over-extension and shrinkage Turns out that: for discontinuities perpendicular to epipolar lines And: for discontinuities parallel to epipolar lines

14. 14 Random Dot Stereogram Experiments

15. 15 Random Dot Stereogram Experiments

16. 16 Offset Windows  Equivalent to using min nearby cost  Result: loss of depth accuracy

17. 17 Discontinuity Detection Use offset windows only where appropriate Bi-modal distribution of SSD Pixel of interest different than mode within window

18. 18 Compact Windows [Veksler CVPR03]: Adapt windows size based on: Average matching error per pixel Variance of matching error Window size (to bias towards larger windows) Pick window that minimizes cost

19. 19 Integral Image Sum of shaded part Compute an integral image for pixel dissimilarity at each possible disparity AC DB Shaded area = A+D-B-C Independent of size

20. 20 Results using Compact Windows

21. 21 Rod-shaped filters Instead of square windows aggregate cost in rod-shaped shiftable windows [Kim CVPR05] Search for one that minimizes the cost (assume that it is an iso-disparity curve) Typically use 36 orientations

22. 22 Locally Adaptive Support Apply weights to contributions of neighboring pixels according to similarity and proximity [Yoon CVPR05]

23. 23 Locally Adaptive Support Similarity in CIE Lab color space: Proximity: Euclidean distance Weights:

24. 24 Locally Adaptive Support: Results Locally Adaptive Support

25. 25 Locally Adaptive Support: Results

26. Occlusions (Slide from Pascal Fua)

27. Exploiting scene constraints

28. Ordering constraint 1 2 3 4,5 6 1 2,3 4 5 6 2 1 3 4,5 6 1 2,3 4 5 6 surface slice surface as a path occlusion right occlusion left

29. Uniqueness constraint In an image pair each pixel has at most one corresponding pixel In general one corresponding pixel In case of occlusion there is none

30. Disparity constraint surface slice surface as a path bounding box disparity band use reconstructed features to determine bounding box constant disparity surfaces

31. Stereo matching Optimal path (dynamic programming ) Similarity measure (SSD or NCC) Constraints epipolar ordering uniqueness disparity limit Trade-off Matching cost (data) Discontinuities (prior) Consider all paths that satisfy the constraints pick best using dynamic programming

32. Hierarchical stereo matching Downsampling (Gaussian pyramid) Disparity propagation Allows faster computation Deals with large disparity ranges

33. Disparity map image I(x,y) image I´(x´,y´) Disparity map D(x,y) (x´,y´)=(x+D(x,y),y)

34. Example: reconstruct image from neighboring images

36. 36 Semi-global optimization Optimize: E=E data +E(|D p -D q |=1)+E(|D p -D q |>1) [Hirshmüller CVPR05] Use mutual information as cost NP-hard using graph cuts or belief propagation (2-D optimization) Instead do dynamic programming along many directions Don’t use visibility or ordering constraints Enforce uniqueness Add costs

37. 37 Results of Semi-global optimization

38. 38 Results of Semi-global optimization No. 1 overall in Middlebury evaluation (at 0.5 error threshold as of Sep. 2006)

39. Energy minimization (Slide from Pascal Fua)

40. Graph Cut (Slide from Pascal Fua) (general formulation requires multi-way cut!)

41. (Boykov et al ICCV‘99) (Roy and Cox ICCV‘98) Simplified graph cut

42. Belief Propagation per pixel +left +right +up +down 2 3 4 5 20... subsequent iterations (adapted from J. Coughlan slides) first iteration Belief of one node about another gets propagated through messages (full pdf, not just most likely state)

44. Stereo matching with general camera configuration

45. Image pair rectification

46. Planar rectification Bring two views to standard stereo setup (moves epipole to  ) (not possible when in/close to image) ~ image size (calibrated) Distortion minimization (uncalibrated)

48. Polar re-parameterization around epipoles Requires only (oriented) epipolar geometry Preserve length of epipolar lines Choose  so that no pixels are compressed original image rectified image Polar rectification (Pollefeys et al. ICCV’99) Works for all relative motions Guarantees minimal image size

49. polar rectification planar rectification original image pair

50. Example: Béguinage of Leuven Does not work with standard Homography-based approaches

51. Example: Béguinage of Leuven

52. General iso-disparity surfaces ( Pollefeys and Sinha, ECCV’04) Example: polar rectification preserves disp. Application: Active vision Also interesting relation to human horopter

53. Stereo camera configurations (Slide from Pascal Fua)

54. Multi-camera configurations Okutami and Kanade (illustration from Pascal Fua)

55. Multi-baseline, multi-resolution At each depth, baseline and resolution selected proportional to that depth Allows to keep depth accuracy constant! Variable Baseline/Resolution Stereo (Gallup et al., CVPR08)

56. Variable Baseline/Resolution Stereo: comparison

57. Multi-view depth fusion Compute depth for every pixel of reference image Triangulation Use multiple views Up- and down sequence Use Kalman filter (Koch, Pollefeys and Van Gool. ECCV‘98) Allows to compute robust texture

58. Plane-sweep multi-view matching Simple algorithm for multiple cameras no rectification necessary doesn’t deal with occlusions Collins’96; Roy and Cox’98 (GC); Yang et al.’02/’03 (GPU)

59. Next class: structured light and active scanning