1. Thanks to Richard Szeliski and George Bebis for the use of some slides
Stereopsis
Thanks to Richard Szeliski and George Bebis for the use of some slides

2. Mark Twain at Pool Table", no date, UCR Museum of Photography

3. Woman getting eye exam during immigration procedure at Ellis Island, c
Woman getting eye exam during immigration procedure at Ellis Island, c , UCR Museum of Phography

4. Why Stereo Vision? 2D images project 3D points into 2D:
P’=Q’ P Q
3D Points on the same viewing line have the same 2D image:
2D imaging results in depth information loss

5. Stereo
Assumes (two) cameras.
Known positions.
Recover depth.

6. Recovering Depth Information:Q
P’1
P’2=Q’2
Q’1 O2 O1
Depth can be recovered with two images and triangulation.

7. Finding Correspondences:Q P
P’1
P’2 Q’2
Q’1 O2 O1

8. Finding Correspondences:

9. Correspondence problem
Multiple hypotheses satisfy the epipolar constraint. Which one is correct?

10. 3D Reconstruction P P’1 P’2 O2 O1
We must solve the correspondence problem first!

11. Stereo correspondence
Determine Pixel Correspondence
Pairs of points that correspond to same scene point
epipolar plane
epipolar line
epipolar line
Epipolar Constraint
Reduces correspondence problem to 1D search along conjugate epipolar lines
(Seitz)

12. Simplest Case Image planes of cameras are parallel.
Focal points are at same height.
Focal lengths same.
Then, epipolar lines are horizontal scan lines.

13. Epipolar Geometry for Parallel Cameras
Epipoles are at infinite
Epipolar lines are parallel to the baseline

14. We can always achieve this geometry with image rectification
Image Reprojection
reproject image planes onto common plane parallel to line between optical centers
Notice, only focal point of camera really matters
(Seitz)

15. Stereo: epipolar geometry
for two images (or images with collinear camera centers), can find epipolar lines
epipolar lines are the projection of the pencil of planes passing through the centers
Rectification: warping the input images (perspective transformation) so that epipolar lines are horizontal

16. Rectification
Project each image onto same plane, which is parallel to the epipole
Resample lines (and shear/stretch) to place lines in correspondence, and minimize distortion
[Zhang and Loop, MSR-TR-99-21]

17. Example
courtesy of Marc Pollefeys

18. Epipolar Line Example
courtesy of Marc Pollefeys

19. Rectification

20. Rectification

21. Rectification Epipolar lines are colinear and parallel to baseline
Epipolar lines are at arbitrary
locations and orientations

22. Rectification (cont’d)
Rectification is a transformation which makes pairs of conjugate epipolar lines become collinear and parallel to the horizontal axis (i.e., baseline)
Searching for corresponding points becomes much simpler for the case of rectified images ul ur
rectification

23. Rectification (cont’d)
Disparities between the
images are in the x-direction
only (i.e., no y disparity)

24. Rectification: Example
before
rectification
after
rectification

25. Rectification (cont’d)
Main steps
(assuming knowledge of the extrinsic/intrinsic stereo parameters):
(1) Rotate the left camera so
that the epipolar lines become
parallel to the horizontal axis
(i.e., epipole is mapped to infinity).

26. Rectification (cont’d)
(2) Apply the same rotation to
the right camera to recover the
original geometry.
(3) Align right camera with left camera using rotation R.
(4) Adjust scale in both camera frames.

27. Rectification (cont’d)
Consider step (1) only (i.e., other steps are easy):
Construct a coordinate system (e1, e2, e3) centered at Ol .
Aligning it with image plane coordinate system.

28. Rectification (cont’d)
(1.1) e1 is a unit vector along the vector T (baseline)

29. Rectification (cont’d)
(1.2) e2 must be perpendicular to e1 (i.e., cross product of e1 with the optical axis)
z=[0,0,1]T

30. Rectification (cont’d)
(1.3) choose e3 as the cross product of e1 and e2

31. Rectification (cont’d)
The rotation matrix that maps the left epipole to infinity is the
transformation that aligns (e1, e2, e3) with (i ,j, k):

32. Then given Z, we can compute X and Y.
Let’s discuss reconstruction with this geometry before correspondence, because it’s much easier. blackboard P Z
Disparity: xl xr f pl pr Ol Or T
Then given Z, we can compute X and Y.
T is the stereo baseline
d measures the difference in retinal position between corresponding points

33. Correspondence: What should we match?
Objects?
Edges?
Pixels?
Collections of pixels?

34. Julesz: had huge impact because it showed that recognition not needed for stereo.

36. Correspondence: Epipolar constraint.

37. Correspondence Problem
Two classes of algorithms:
Correlation-based algorithms
Produce a DENSE set of correspondences
Feature-based algorithms
Produce a SPARSE set of correspondences

38. Correspondence: Photometric constraint
Same world point has same intensity in both images.
Lambertian fronto-parallel
Issues:
Noise
Specularity
Foreshortening

39. Using these constraints we can use matching for stereo
Improvement: match windows
For each pixel in the left image
For each epipolar line
compare with every pixel on same epipolar line in right image
pick pixel with minimum match cost
This will never work, so:

40. Comparing Windows: ? = g f Most popular
For each window, match to closest window on epipolar line in other image.

41. Minimize
Sum of Squared
Differences
Maximize
Cross correlation
It is closely related to the SSD:

42. Similarity Constraint
Left
Right
scanline
Matching cost
disparity
Slide a window along the right scanline and compare its contents with the reference window in the left image
Matching cost: SSD or normalized correlation

43. Similarity Constraint
Left
Right
scanline
SSD

44. Similarity Constraint
Left
Right
scanline
Norm. corr

45. Window size W = 3 W = 20 Effect of window size
Better results with adaptive window
T. Kanade and M. Okutomi, A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment,, Proc. International Conference on Robotics and Automation, 1991.
D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffusion. International Journal of Computer Vision, 28(2): , July 1998
smaller window: more detail, more noise
bigger window: less noise, more detail
(Seitz)

47. Stereo results Data from University of Tsukuba Scene Ground truth
(Seitz)

48. Results with window correlation
Window-based matching
(best window size)
Ground truth
(Seitz)

49. Results with better method
State of the art method
Boykov et al., Fast Approximate Energy Minimization via Graph Cuts,
International Conference on Computer Vision, September 1999.
Ground truth
(Seitz)

50. Ordering constraint Usually, order of points in two images is same.
Is this always true?

51. This enables dynamic programming.
If we match pixel i in image 1 to pixel j in image 2, no matches that follow will affect which are the best preceding matches.
Example with pixels (a la Cox et al.).

52. Other constraints
Smoothness: disparity usually doesn’t change too quickly.
Unfortunately, this makes the problem 2D again.
Solved with a host of graph algorithms, Markov Random Fields, Belief Propagation, ….
Uniqueness constraint (each feature can at most have one match
Occlusion and disparity are connected.

53. Feature-based Methods
Conceptually very similar to Correlation-based methods, but:
They only search for correspondences of a sparse set of image features.
Correspondences are given by the most similar feature pairs.
Similarity measure must be adapted to the type of feature used.

54. Feature-based Methods:
Features most commonly used:
Corners
Similarity measured in terms of:
surrounding gray values (SSD, Cross-correlation)
location
Edges, Lines
orientation
contrast
coordinates of edge or line’s midpoint
length of line

55. Example: Comparing lines
ll and lr: line lengths
ql and qr: line orientations
(xl,yl) and (xr,yr): midpoints
cl and cr: average contrast along lines
wl wq wm wc : weights controlling influence
The more similar the lines, the larger S is!

56. Summary
First, we understand constraints that make the problem solvable.
Some are hard, like epipolar constraint.
Ordering isn’t a hard constraint, but most useful when treated like one.
Some are soft, like pixel intensities are similar, disparities usually change slowly.
Then we find optimization method.
Which ones we can use depend on which constraints we pick.

57. SEGMENTATION AND STEREO

58. Compositional Correspondence+Segmentation Algorithm
(D) Find best disparity for each point P(x,y)
(A) Match
(C) Find sizes
(B) Connected components
: find matching pixels for different shifts
P(x,y)
No match 20 10 78 7 9 8 5 41 27 47 25 78 41 20
...
Shift = 0
Shift = 1
Shift = 2
Shift = n
...
For each point P(x,y), compare the sizes of the connected components containing it.

59. Results: Stereo Matching (IJCV’05)

60. Introducing Shape (Slanted surfaces, CVPR’04)z
Image plane x

61. Slant: Sampling, Correspondence, and Occlusions (CVPR’04)
Slanted surfaces are sampled differently.
N to M pixel correspondence.
Can no longer use uniqueness constraint to find occlusions.
Slant, Sampling, Correspondence, Segmentation, Occlusions are all interconnected !

62. Compositional estimation of Shape (IJCV’05)
Joint
Estimation
xRIGHT = M xLEFT + D
Search over slants and shifts!

63. Results - Horizontally slanted object
Left image
Right image
Graph cuts
(Kolmogorov-Zabih
ICCV’01)
Our result

64. Results - Vertically slanted object
Left image
Right image
Graph cuts
(Kolmogorov-Zabih
ICCV’01)
Our result

65. Contrast Invariance using Gabor Filters (ICRA’04)
Orientation
Scale
Real part
(4 scales, 4 orientations)
Imag. part
Filter both images using complex Gabor filters.
Find phase differences in each channel, ignoring amplitude.
Compute local matching.

66. Results (ICRA’04, IJCV’06 Special Issue)
Left images
Right images
Disparity Maps