2. Computer Vision Spring 2010 15-385,-685 Instructor: S. Narasimhan WH 5409 T-R 10:30am – 11:50am Lecture #15

3. Binocular Stereo Lecture #15

4. Recovering 3D from Images How can we automatically compute 3D geometry from images? –What cues in the image provide 3D information?

5. Shading Visual Cues for 3D Merle Norman Cosmetics, Los Angeles

6. Visual Cues for 3D Shading Texture The Visual Cliff, by William Vandivert, 1960

7. Visual Cues for 3D From The Art of Photography, Canon Shading Texture Focus

8. Visual Cues for 3D Shading Texture Focus Motion

9. Visual Cues for 3D Others: –Highlights –Shadows –Silhouettes –Inter-reflections –Symmetry –Light Polarization –... Shading Texture Focus Motion Shape From X X = shading, texture, focus, motion,... We’ll focus on the motion cue

10. Stereo Reconstruction The Stereo Problem –Shape from two (or more) images –Biological motivation knowncameraviewpoints

11. Why do we have two eyes? Cyclope vs. Odysseus

12. 1. Two is better than one

13. 2. Depth from Convergence

14. 3. Depth from binocular disparity Sign and magnitude of disparity P: converging point C: object nearer projects to the outside of the P, disparity = + F: object farther projects to the inside of the P, disparity = -

15. Binocular Stereo

17. Disparity and Depth scene left image right image baseline Assume that we know corresponds to From perspective projection (define the coordinate system as shown above)

18. Disparity and Depth is the disparity between corresponding left and right image points disparity increases with baseline b inverse proportional to depth scene left image right image baseline

19. field of view of stereo Vergence one pixel uncertainty of scenepoint Optical axes of the two cameras need not be parallel Field of view decreases with increase in baseline and vergence Accuracy increases with baseline and vergence

20. Binocular Stereo scene point optical center image plane

21. Binocular Stereo Basic Principle: Triangulation –Gives reconstruction as intersection of two rays –Requires calibration point correspondence

22. Stereo Correspondence Determine Pixel Correspondence –Pairs of points that correspond to same scene point Epipolar Constraint –Reduces correspondence problem to 1D search along conjugate epipolar lines –Java demo: http://www.ai.sri.com/~luong/research/Meta3DViewer/EpipolarGeo.html http://www.ai.sri.com/~luong/research/Meta3DViewer/EpipolarGeo.html epipolar plane epipolar line

23. Stereo Image Rectification

24. reproject image planes onto a common plane parallel to the line between optical centers pixel motion is horizontal after this transformation  C. Loop and Z. Zhang. Computing Rectifying Homographies for Stereo Vision. IEEE Conf. Computer Vision and Pattern Recognition, 1999.Computing Rectifying Homographies for Stereo Vision Stereo Image Rectification Details in next lecture

25. Stereo Rectification

26. Basic Stereo Algorithm For each epipolar line For each pixel in the left image compare with every pixel on same epipolar line in right image pick pixel with minimum match cost Improvement: match windows This should look familiar... Correlation, Sum of Squared Difference (SSD), etc.

27. Size of Matching window –Smaller window Good/bad ? –Larger window Good/bad ? W = 3W = 20 Better results with adaptive window T. Kanade and M. Okutomi, A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment,, Proc. International Conference on Robotics and Automation, 1991.A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffusion. International Journal of Computer Vision, 28(2):155-174, July 1998Stereo matching with nonlinear diffusion Effect of window size

28. Stereo Results Ground truthScene –Data from University of Tsukuba

29. Results with Window Search Window-based matching (best window size) Ground truth

30. Better methods exist... State of the art method Boykov et al., Fast Approximate Energy Minimization via Graph Cuts,Fast Approximate Energy Minimization via Graph Cuts International Conference on Computer Vision, September 1999. Ground truth

31. Stereo Example input image (1 of 2) [Szeliski & Kang ‘95] depth map 3D rendering

32. Stereo Example left imageright image depth map H. Tao et al. “Global matching criterion and color segmentation based stereo”Global matching criterion and color segmentation based stereo

33. Stereo Example H. Tao et al. “Global matching criterion and color segmentation based stereo”Global matching criterion and color segmentation based stereo

34. Stereo Matching Features vs. Pixels? –Do we extract features prior to matching? Julesz-style Random Dot Stereogram

35. Range Scanning and Structured Light

36. Next Class Binocular Stereo (relative and absolute orientation) Reading: Horn, Chapter 13.