1. Correspondence and Stereopsis

2. Introduction Disparity – Informally: difference between two pictures – Allows us to gain a strong sense of depth Stereopsis – Ability to perceive depth from disparity Goal of this chapter – Design algorithms that mimic stereopsis

3. Applications of Stereopsis Visual robot navigation Cartography Aerial reconnaissance Close-range photogrammetry Image segmentation for object recognition

4. Stereo Vision Two processes – Binocular fusion of features observed by the eyes – Reconstruction of their three-dimensional preimage

5. Stereo Vision – Easy Case 1 single point being observed – The preimage can be found at the intersection of the rays from the focal points to the image points

6. Stereo Vision – Hard Case Many points being observed – Need some method to establish correct correspondences

7. Components of Stereo Vision Systems Camera calibration: previous lectures Image rectification: simplifies the search for correspondences Correspondence: which item in the left image corresponds to which item in the right image Reconstruction: recovers 3-D information from the 2-D correspondences

8. Epipolar Geometry Epipolar constraint: corresponding points must lie on conjugated epipolar lines – Search for correspondences becomes a 1-D problem

9. Image Rectification Corresponding epipolar lines become collinear

10. Image Rectification (cont.) Not equivalent to rotation The lines through the centers become parallel to each other, and the epipoles move to infinity

11. Image Rectification (cont.) Given extrinsic parameters T and R (relative position and orientation of the two cameras) – Rotate the left camera about the projection center so that the the epipolar lines become parallel to the horizontal axis – Apply the same rotation to the right camera – Rotate the right camera by R – Adjust the scale in both camera reference frames

12. Image Rectification (cont.) Formal definition of disparity: d=u'–u

13. Correspondence Given an element in the left image, find the corresponding element in the right image Classes of methods – Correlation-based – Feature-based

14. Correlation-Based Correspondence Input: rectified stereo pair and a point (u,v) in the first image Method: – Associate a window of size p=(2m+1)(2n+1) centered in (u,v) and form the vector w(u,v) in R p – For each potential match (u+d,v) in the second image, compute w' and the normalized correlation between w and w'

15. Correlation-Based Correspondence (cont.) Main problem: – Implicitly assume that the observed surface is locally parallel to the two image planes – Alleviated by computing an initial disparity and using it to warp the correlation windows to compensate for unequal amounts of foreshortening Other problems: – Not robust against noise – Similar pixels may not correspond to physical features

16. Feature-Based Correspondence Main idea: physically-significant features should be preferred to matches between raw pixel intensities Instead of correlation-like measures, use a measure of the distance between feature descriptors Typical features: points, lines, and corners Example: Marr-Poggio-Grimson algorithm

17. Marr-Poggio-Grimson Algorithm Convolve images with Laplacian of Gaussian filters with standard deviations  1 <  2 <  3 <  4 Find zero crossings of the Laplacian along horizontal scanlines of the filtered images For each , match zero crossings with same parity and similar orientations in a [–w ,w  ] disparity range, with

18. Marr-Poggio-Grimson Algorithm (cont.) Use disparities found at larger scales to control eye vergence and cause unmatched regions at smaller scales to come into correspondence

19. Marr-Poggio-Grimson algorithm (cont.)

21. Ordering Constraint The order of matching image features along a pair of epipolar lines is (usually) the inverse of the order of the corresponding surface attributes along the curve where the epipolar plane intersects the object's boundary

22. Ordering Constraint (cont.) May not be satisfied in real scenes due to occlusion Still useful to devise efficient algorithms relying on dynamic programming to establish stereo correspondences

23. Reconstruction Given pair of image points p and p', and focal points O and O', find preimage P In theory: find P by intersecting the rays R=Op and R'=Op' In practice: R and R' won't actually intersect due to calibration and feature localization errors

24. Reconstruction Approaches Geometric – Construct the line segment perpendicular to R and R' that intersects both rays and take its mid-point

25. Reconstruction Approaches (cont.) Algebraic (linear) – Write down the projection equations – The resulting linear system is overconstrained – Solve it by linear least-squares

26. Reconstruction Approaches (cont.) Algebraic (non-linear) – Find the point Q that minimizes d 2 (p,q)+d 2 (p',q') by non-linear least-squares – Reconstructions obtained by the previous methods can be used as initial guesses for the optimization