1. Computer Vision, Robert Pless
our goal is to understand the process of stereo vision.
Last time, we studied the “Essential” Matrix, which describe where a point in one image may appear in a second image.
Computer Vision, Robert Pless

2. Computer Vision, Robert Pless
Rectification
Need to rotate each camera so the epipole is along the y-axis. (0,+/-1,0)
Is this unique?
Computer Vision, Robert Pless

3. Stereo Image Rectification
Computer Vision, Robert Pless

4. Computer Vision, Robert Pless

5. Stereo Image Rectification
reproject image planes onto a common
plane parallel to the line between optical centers
pixel motion is horizontal after this transformation
C. Loop and Z. Zhang. Computing Rectifying Homographies for Stereo Vision. IEEE Conf. Computer Vision and Pattern Recognition, 1999.
Computer Vision, Robert Pless

6. Computer Vision, Robert Pless
Stereo Rectification
Computer Vision, Robert Pless

7. Computer Vision, Robert Pless
Rectification
Computer Vision, Robert Pless

8. Basic Stereo Algorithm
Improvement: match windows
This should look familar...
Correlation, Sum of Squared Difference (SSD), etc.
For each pixel in the left image
For each epipolar line
compare with every pixel on same epipolar line in right image
pick pixel with minimum match cost
Computer Vision, Robert Pless

9. Computer Vision, Robert Pless
Then, finally, P Z
Disparity: xl xr f pl pr Ol Or b
Then given Z, we can compute X and Y.
b is the stereo baseline
d measures the difference in retinal position between corresponding points
Computer Vision, Robert Pless

10. Using these constraints we can use matching for stereo
Improvement: match windows
For each pixel in the left image
For each epipolar line
compare with every pixel on same epipolar line in right image
pick pixel with minimum match cost
This will never work, so:
Computer Vision, Robert Pless

11. Correspondence: Photometric constraint
Same world point has same intensity in both images.
Lambertian fronto-parallel
Issues:
Noise
Specularity
Foreshortening
Computer Vision, Robert Pless

12. Computer Vision, Robert Pless= ? f g
Comparing Windows:
Most
popular
For each window, match to closest window on epipolar line in other image.
Computer Vision, Robert Pless

13. Computer Vision, Robert Pless
Window size
W = 3
W = 20
Effect of window size
Better results with adaptive window
T. Kanade and M. Okutomi, A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment,, Proc. International Conference on Robotics and Automation, 1991.
D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffusion. International Journal of Computer Vision, 28(2): , July 1998
smaller window: more detail, more noise
bigger window: less noise, more detail
Computer Vision, Robert Pless

14. Computer Vision, Robert Pless
Stereo results
Data from University of Tsukuba
Scene
Ground truth
Computer Vision, Robert Pless

15. Results with window correlation
Window-based matching
(best window size)
Ground truth
Computer Vision, Robert Pless

16. Results with better method
State of the art method
Boykov et al., Fast Approximate Energy Minimization via Graph Cuts,
International Conference on Computer Vision, September 1999.
Ground truth
Computer Vision, Robert Pless

17. Computer Vision, Robert Pless
Stereo Results
Trinocular stereo system available from Point Gray Research for $5K (circa ’97)
Computer Vision, Robert Pless

18. Computer Vision, Robert Pless
Stereo Example
[Szeliski & Kang ‘95]
depth map
3D rendering
input image (1 of 2)
Computer Vision, Robert Pless

19. Computer Vision, Robert Pless
Stereo Example
left image
right image
depth map
H. Tao et al. “Global matching criterion and color segmentation based stereo”
Computer Vision, Robert Pless

20. Computer Vision, Robert Pless
Stereo Example
Computer Vision, Robert Pless
H. Tao et al. “Global matching criterion and color segmentation based stereo”

21. What are these additional constraints?
corresponding points along corresponding epipolar lines
Computer Vision, Robert Pless

22. Order !
Once the epipolar geometry is known, we can increase constraints on potential correspondences further...
corresponding points along corresponding epipolar lines
3d points giving rise to those images
Computer Vision, Robert Pless

23. Order !
Out-of-order points indicate “seeing around” an object, and generally doesn’t occur ...
corresponding points along corresponding epipolar lines
3d points giving rise to those images
Can you see the dynamic programming problem here?
Computer Vision, Robert Pless

24. Computer Vision, Robert Pless
Correspondence
It is fundamentally ambiguous, even with stereo constraints
Ordering constraint…
…and its failure
Computer Vision, Robert Pless

25. Search Over Correspondences
Occluded Pixels
Left scanline
Right scanline
Dis-occluded Pixels
Three cases:
Sequential – cost of match
Occluded – cost of no match
Disoccluded – cost of no match
Computer Vision, Robert Pless

26. Stereo Matching with Dynamic Programming
Occluded Pixels
Left scanline
Start
Dynamic programming yields the optimal path through grid. This is the best set of matches that satisfy the ordering constraint
Dis-occluded Pixels
Right scanline
End
Computer Vision, Robert Pless

27. Stereo Matching with Dynamic Programming
Occluded Pixels
Left scanline
Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path.
Dis-occluded Pixels
Right scanline
Terminal
Computer Vision, Robert Pless

28. Stereo Matching with Dynamic Programming
Occluded Pixels
Left scanline
Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path.
Dis-occluded Pixels
Right scanline
Terminal
Computer Vision, Robert Pless

29. Stereo Matching with Dynamic Programming
Occluded Pixels
Left scanline
Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path.
Dis-occluded Pixels
Right scanline
Terminal
Computer Vision, Robert Pless

30. Stereo Matching with Dynamic Programming
Occluded Pixels
Left scanline
Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path.
Dis-occluded Pixels
Right scanline
Terminal
Computer Vision, Robert Pless

31. Stereo Matching with Dynamic Programming
Occluded Pixels
Left scanline
Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path.
Dis-occluded Pixels
Right scanline
Terminal
Computer Vision, Robert Pless

32. Stereo Matching with Dynamic Programming
Occluded Pixels
Left scanline
Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path.
Dis-occluded Pixels
Right scanline
Terminal
Computer Vision, Robert Pless

33. Stereo Matching with Dynamic Programming
Occluded Pixels
Left scanline
Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path.
Dis-occluded Pixels
Right scanline
Terminal
Computer Vision, Robert Pless

34. Computing Correspondence
Another approach is to match edges rather than windows of pixels:
Which method is better?
Edges tend to fail in dense texture (outdoors)
Correlation tends to fail in smooth featureless areas
Computer Vision, Robert Pless

35. Computing Correspondences
Both methods fail for smooth surfaces
There is currently no good solution to the correspondence problem
Computer Vision, Robert Pless

36. History and Philosophy of human stereo
Features vs. Pixels?
Do we extract features prior to matching?
Julesz-style Random Dot Stereogram
Computer Vision, Robert Pless

37. Computer Vision, Robert Pless
Human abilities
Computer Vision, Robert Pless

38. Computer Vision, Robert Pless
Human abilities
Computer Vision, Robert Pless

39. Computer Vision, Robert Pless
Assignment: Stereo In or Stereo Out (2 options)
Write a program that takes as input a rectified stereo pair, and creates the 3D model of the scene.
Show your program output on:
Write a program that takes as input a depth map, and creates an auto-stereogram that gives that depth map.
Google search for “depth map”
Computer Vision, Robert Pless