1. Introduction to Computer Vision 3D Vision Topic 9 Stereo Vision (I) CMPSCI 591A/691A CMPSCI 570/670

2. Introduction to Computer Vision 2D to 3D Inference n Observations l Objects are mostly 3D l Images are 2D arrays of intensity, color values, etc. l 3D depth information is not explicitly encoded in images (it is explicitly recorded in range images) n However, 2D analysis implicitly uses 3D info l 3D structures are generally not random n coherency in motion l Man-made objects are of regular shapes and boundaries n straight lines and smooth curves in images n Explicit 3D information can be recovered using 2D shape cues l disparities in stereo l shading change due to orientation l texture gradient due to view point change etc.

3. Introduction to Computer Vision View Images as “windows” into the 3D world

4. Introduction to Computer Vision Shape Inference Techniques

5. Introduction to Computer Vision Stereo n The Stereo Problem l Reconstruct scene geometry from two or more calibrated images n Basic Principle: Triangulation l Gives reconstruction as intersection of two rays l Requires point correspondence n This is the hard part

6. Introduction to Computer Vision Stereo Vision n Problem l Infer 3D structure of a scene from two or more images taken from different viewpoints n Two primary Sub-problems l Correspondence problem (stereo match) -> disparity map n Similarity instead of identity n Occlusion problem: some parts of the scene are visible in one eye only l Reconstruction problem -> 3D n What we need to know about the cameras’ parameters n Often a stereo calibration problem n Lectures on Stereo Vision l Stereo Geometry – Epipolar Geometry (*) l Correspondence Problem (*) – Two classes of approaches l 3D Reconstruction Problems – Three approaches

7. Introduction to Computer Vision A Stereo Pair n Problems l Correspondence problem (stereo match) -> disparity map l Reconstruction problem -> 3D CMU CIL Stereo Dataset : Castle sequence http://www-2.cs.cmu.edu/afs/cs/project/cil/ftp/html/cil-ster.html ? 3D?

8. Introduction to Computer Vision More Images…

9. Introduction to Computer Vision More Images…

10. Introduction to Computer Vision More Images…

11. Introduction to Computer Vision More Images…

12. Introduction to Computer Vision More Images…

13. Introduction to Computer Vision Stereo View Left View Right View Disparity

14. Introduction to Computer Vision Stereo Disparity n The separation between two matching objects is called the stereo disparity. n Disparity is measured in pixels and can be positive or negative (conventions differ). It will vary across the image.

15. Introduction to Computer Vision Disparity and Depth n If the cameras are pointing in the same direction, the geometry is simple. l b is the baseline of the camera system, l Z is the depth of the object, l d is the disparity (left x minus right x) and l f is the focal length of the cameras. n Then the unknown depth is given by b Z = f b d

16. Introduction to Computer Vision Basic Stereo Configuration

17. Introduction to Computer Vision Parallel Cameras n Disparity is inversely proportional to depth—so stereo is most accurate for close objects. n Once we have found depth, the other coordinates in 3-D follow easily — e.g. taking either one of the images: n where x is the image coordinate, and likewise for Y.

18. Introduction to Computer Vision Converging Cameras n This is the more realistic case. n The depth at which the cameras converge, Z0, is the depth at which objects have zero disparity. l Finding Z0 is part of stereo calibration. n Closer objects have convergent disparity (numerically positive) and further objects have divergent disparity (numerically negative). Object at depth Z

19. Introduction to Computer Vision The Correspondence Problem n To measure disparity, we first have to find corresponding points in the two images. n This turns out not to be easy. n Our own visual systems can match at a low level, as shown by random-dot stereograms, in which the individual images have no structure above pixel scale, but which when fused show a clear 3-D shape.

20. Introduction to Computer Vision Stereo Matching n Stereo matchers need to start from some assumptions. l Corresponding image regions are similar. l A point in one image may only match a single point in the other image. l If two matched features are close together in the images, then in most casestheir disparities will be similar, because the environment is made of continuous surfaces separated by boundaries. n Many matching methods exist. The basic distinction is between l feature-based methods which start from image structure extracted by preprocessing; and l correlation-based methods which start from individual grey- levels.

21. Introduction to Computer Vision Feature Based Methods n Extract feature descriptions: regions, lines, …. n Pick a feature in the left image. n Take each feature in the right image in turn (or just those close to the epipolar line), and measure how different it is from the original feature: Size, aspect ratio, average grey level etc.

22. Introduction to Computer Vision Feature-Based Methods n S is a measure of similarity l w0 etc. are weights, and l the other symbols are different measures of the feature in the right and left images, such as length, orientation, average grey level and so on. n Choose the right-image feature with the largest value of S as the best match for the original left-image feature. n Repeat starting from the matched feature in the right image, to see if we achieve consistency.

23. Introduction to Computer Vision Feature Based Methods n It is possible to use very simple features (just points, in effect) if the constraint that the disparity should vary smoothly is taken into account. n Feature-based methods give a sparse set of disparities — disparities are only found at feature positions.

24. Introduction to Computer Vision Correlation Based Methods n Take a small patch of the left image as a mask and convolve it with the part of the right image close to the epipolar line. l Over an area n The peak of the convolution output gives the position of the matching area of the right image, and hence the disparity of the best match.

25. Introduction to Computer Vision section of left image convolve Convolution peak at position of corresponding patch in right image Correlation-Based Methods n Convolution-based methods can give a dense set of disparities — disparities are found for every pixel. n These methods can be very computationally intensive, but can be done efficiently on parallel hardware. Where to search?

26. Introduction to Computer Vision Stereo Correspondence n Epipolar Constraint l Reduces correspondence problem to 1D search along conjugate epipolar lines l Stereo rectification: make epipolar lines horizontal

27. Introduction to Computer Vision Correspondence Difficulties n Why is the correspondence problem difficult? l Some points in each image will have no corresponding points in the other image. (1) the cameras might have different fields of view. (2) due to occlusion. n A stereo system must be able to determine the image parts that should not be matched.

28. Introduction to Computer Vision Variations n Scale-space. Methods that exploit scale-space can be very powerful. l Smooth the image using Gaussian kernels l Match each feature with the nearest one in the other image — location will be approximate because of blurring. l Use the disparities found to guide the search for matches in a less smoothed image. n Relaxation. This is important in the context of neural modelling. l Set up possible feature matches in a network. l Construct an energy function that captures the constraints of the problem. l Make incremental changes to the matches so that the energy is steadily reduced.

29. Introduction to Computer Vision n Very precise stereo systems can be made to estimate disparity at sub-pixel accuracies. This is important for industrial inspection and mapping from aerial and satellite images. n In controlled environments, structured light (e.g. stripes of light) can be used to provide easy matches for a stereo system. Other Aspects of Stereo

30. Introduction to Computer Vision Structured Light n Structured lighting l Feature-based methods are not applicable when the objects have smooth surfaces (i.e., sparse disparity maps make surface reconstruction difficult). l Patterns of light are projected onto the surface of objects, creating interesting points even in regions which would be otherwise smooth. l Finding and matching such points is simplified by knowing the geometry of the projected patterns.

31. Introduction to Computer Vision n http://graphics.stanford.edu/projects/mich/ Stanford’s Digital Michaelangelo Project maximum height of gantry: 7.5 meters weight including subbase: 800 kilograms

32. Introduction to Computer Vision Stanford’s Digital Michaelangelo Project 480 individually aimed scans 2 billion polygons 7,000 color images 32 gigabytes 30 nights of scanning 1,080 man-hours 22 people

33. Introduction to Computer Vision Correspondence Problems n Which point matches with which point?

34. Introduction to Computer Vision Difficulties n Multiple matches are always likely l Simple features (e.g., black dots) n large number of potential matches n precise disparity n Complex features (e.g., polygons) l small number of potential matches l less precise disparity

35. Introduction to Computer Vision Trinocular Stereo From H. G. Wells, War of the Worlds

36. Introduction to Computer Vision Three Camera Stereo n A powerful way of eliminate spurious matches l Hypothesize matches between A & B l Matches between A & C on green epipolar line l Matches between B & C on red epipolar line l There better be something at the intersection