1. Stereo and Epipolar geometry
Jana Kosecka
CS223b

2. Previously Image Primitives (feature points, lines, contours) Today:
How to match primitives between two (multiple) views)
Goals: 3D reconstruction, recognition
Stereo matching and reconstruction (canonical configuration)
Epipolar Geometry (general two view setting)
CS223b

3. 2D images project 3D points into 2D:
Why Stereo Vision?
2D images project 3D points into 2D: P Q
P’=Q’ O
center of projection
3D points on the same viewing line have the same 2D image:
2D imaging results in depth information loss
CS223b

4. Canonical Stereo Configuration
Assumes (two) cameras
Known positions and focal lengths
Recover depth P f O1 O2 T
CS223b

5. Depth can be recovered by triangulation
Prerequisite: Finding Correspondences
Given an image point in left image, the (corresponding)
point in the right image, is the point which is the projection
of the same 3-D point
CS223b

6. Random dot stereograms
B. Julesz: Showed that correspondence is not needed for stereo.
CS223b

7. Autostereograms Depth perception from one image
Viewing trick the brain by focusing at the plane behind - match can be established perception of 3D
CS223b

8. Correspondence Problem
Two classes of algorithms:
Correlation-based algorithms
Produce a DENSE set of correspondences
Feature-based algorithms
Produce a SPARSE set of correspondences
CS223b

9. Stereo – Photometric Constraint
Same world point has same intensity in both images.
Lambertian fronto-parallel
Issues (noise, specularities, foreshortening)
Difficulties – ambiguities, large changes of appearance, due to change
Of viewpoint, non-uniquess
CS223b

10. Stereo Matching What if ?
For each scanline , for each pixel in the left image
compare with every pixel on same epipolar line in right image
pick pixel with minimum match cost
This will never work, so: improvement match windows
CS223b
(slides O. Camps)

11. Comparing Windows: ? = g f Most popular
For each window, match to closest window on epipolar line in other image.
(slides O. Camps)
CS223b

12. Comparing Windows: Minimize Sum of Squared Differences Maximize
Cross correlation
It is closely related to the SSD:
CS223b

13. Region based Similarity Metrics
Sum of squared differences
Normalize cross-correlation
Sum of absolute differences
CS223b

14. NCC score for two widely separated views
CS223b

15. Window size W = 3 W = 20 Better results with adaptive window
Effect of window size
smaller window: more detail, more noise
bigger window: less noise, more detail
Better results with adaptive window
T. Kanade and M. Okutomi, A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment,, Proc. International Conference on Robotics and Automation, 1991.
D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffusion. International Journal of Computer Vision, 28(2): , July 1998
(S. Seitz)
CS223b

16. Data from University of Tsukuba
Stereo results
Data from University of Tsukuba
Scene
Ground truth
(Seitz)
CS223b

17. Results with window correlation
Window-based matching
(best window size)
Ground truth
(Seitz)
CS223b

18. Results with better method
State of the art
Ground truth
Boykov et al., Fast Approximate Energy Minimization via Graph Cuts,
International Conference on Computer Vision, September 1999.
(Seitz)
CS223b

19. More of advanced stereo
Ordering constraint
Dynamic programming
Global optimization – next lecture
CS223b

20. Two view – General Configuration
Motion between the two views is not known
Given two views of the scene
recover the unknown camera
displacement and 3D scene
structure
CS223b

21. Pinhole Camera Imaging Model
3D points
Image points
Perspective Projection
Rigid Body Motion
Rigid Body Motion + Persp. projection
CS223b

22. Rigid Body Motion – Two Views
CS223b

23. 3D Structure and Motion Recovery
Euclidean transformation
Measurements
Corresponding points
unknowns
Find such Rotation and Translation and Depth that the reprojection error is minimized
Two views ~ points
6 unknowns – Motion 3 Rotation, 3 Translation
- Structure 200x3 coordinates
- (-) universal scale
Difficult optimization problem
CS223b

24. Notation
Cross product between two vectors in
where
CS223b

25. Epipolar Geometry Algebraic Elimination of Depth Essential matrix
Image
correspondences
Algebraic Elimination of Depth
[Longuet-Higgins ’81]:
Essential matrix
CS223b

26. Additional constraints
Epipolar Geometry
Epipolar lines
Epipoles
Image
correspondences
Epipolar transfer
Additional constraints
CS223b

27. Characterization of Essential Matrix
special 3x3 matrix
(Essential Matrix Characterization)
A non-zero matrix is an essential matrix iff its SVD:
satisfies: with and
and
CS223b

28. Estimating Essential Matrix
Find such Rotation and Translation that the epipolar error is minimized
Space of all Essential Matrices is 5 dimensional
3 DOF Rotation, 2 DOF – Translation (up to scale !)
Denote
Rewrite
Collect constraints from all points
CS223b

29. Estimating Essential Matrix
Solution
Eigenvector associated with the smallest eigenvalue of
If degenerate configuration
estimated using linear least squares
unstack >
Projection on to Essential Space
(Project onto a space of Essential Matrices)
If the SVD of a matrix is given by
then the essential matrix which minimizes the
Frobenius distance is given by
with
CS223b

30. Pose Recovery from Essential Matrix
There are two relative poses with and
corresponding to a non-zero matrix essential matrix.
Twisted pair ambiguity
CS223b

31. Two view linear algorithm - summary
Solve the LLSE problem:
Solution eigenvector associated with
smallest eigenvalue of
Compute SVD of F recovered from data
E is 5 diml. sub. mnfld. in
8-point linear algorithm
Project onto the essential manifold:
Recover the unknown pose:
CS223b

32. Pose Recovery There are two pairs corresponding to essential matrix .
Positive depth constraint disambiguates the impossible solutions
Translation has to be non-zero
Points have to be in general position
- degenerate configurations – planar points
- quadratic surface
Linear 8-point algorithm
Nonlinear 5-point algorithms yields up to 10 solutions
CS223b

33. 3D Structure Recovery Eliminate one of the scale’s Solve LLSE problem
unknowns
Eliminate one of the scale’s
Solve LLSE problem
Alternatively recover each point depth separately
If the configuration is non-critical, the Euclidean structure of the points and motion of the camera can be reconstructed up to a universal scale.
CS223b

34. Example
Two views
Point Feature Matching
CS223b

35. Sparse Structure Recovery
Example
Epipolar Geometry
Camera Pose
and
Sparse Structure Recovery
CS223b

36. Epipolar Geometry - Planar case
Plane in first camera coordinate frame
Image
correspondences
Planar homography
Linear mapping relating two
corresponding planar points
in two views
CS223b

37. Decomposition of H (into motion and plane normal)
Algebraic elimination of depth
can be estimated linearly
Normalization of
Decomposition of H into 4 solutions
CS223b

38. Uncalibrated Camera calibrated coordinates Linear transformation pixel
CS223b

39. Uncalibrated Epipolar Geometry
Epipolar constraint
Fundamental matrix
CS223b

40. Properties of the Fundamental Matrix
Epipolar lines
Epipoles
Image
correspondences
CS223b

41. Epipolar Geometry for Parallel Cameras
Epipoles are at infinite
Epipolar lines are parallel to the baseline
CS223b

42. Properties of the Fundamental Matrix
A nonzero matrix is a fundamental matrix if has
a singular value decomposition (SVD) with
for some
There is little structure in the matrix except that
CS223b

43. Estimating Fundamental Matrix
Find such F that the epipolar error is minimized
Pixel coordinates
Fundamental matrix can be estimated up to scale
Denote
Rewrite
Collect constraints from all points
CS223b

44. Two view linear algorithm – 8-point algorithm
Solve the LLSE problem:
Solution eigenvector associated with
smallest eigenvalue of
Compute SVD of F recovered from data
Project onto the essential manifold:
cannot be unambiguously decomposed into pose
and calibration
CS223b

45. Projective Reconstruction
Euclidean Motion Cannot be obtained in uncalibrated setting (F cannot be uniquely decomposed into R,T and K matrix)
Can we still say something about 3D ?
Notion of the projective 3D structure
(study of projective geometry)
Euclidean reconstruction
Projective reconstruction
CS223b

46. Euclidean vs Projective reconstruction
Euclidean reconstruction – true metric properties of objects
lenghts (distances), angles, parallelism are preserved
Unchanged under rigid body transformations
=> Euclidean Geometry – properties of rigid bodies under
rigid body transformations, similarity transformation
Projective reconstruction – lengths, angles, parallelism are NOT preserved – we get distorted images of objects – their distorted 3D counterparts --> 3D projective reconstruction
=> Projective Geometry
CS223b

47. Image rectification Given general displacement how to warp the views
Such that epipolar lines are parallel to each other
How to warp it back to canonical configuration
(more details later)
(Seitz)
CS223b

48. Epipolar rectification
Rectified Image Pair
Corresponding epipolar lines are aligned with the scan-lines
Search for dense correspondence is a 1D search
CS223b

49. Epipolar rectification
Rectified Image Pair
CS223b

50. Special Motions – Pure Rotation
Calibrated Two views related by rotation only
Uncalibrated Case
Mapping to a reference view – H can be estimated
Mapping to a cylindrical surface - applications – image mosaics
CS223b

51. Epipolar rectification
Rectified Image Pair
Corresponding epipolar lines are aligned with the scan-lines
Search for dense correspondence is a 1D search
CS223b

52. Using H p’ p Image based rectification (given some points in 3D world)
compute H which would map them into a square
Use H to rectify the entire image
In calibrated case inter-image homography
CS223b

53. Single view modeling in projective setting
Vermeer’s Music Lesson
Planes and distances from the planes can be computed
Single view Rendering (A. Crimisi)
CS223b

54. More on stereo
CS223b

55. Usually, order of points in two images is same. Is this always true?
Ordering constraint
Usually, order of points in two images is same.
Is this always true?
If we match pixel i in image 1 to pixel j in image 2, no matches that follow will affect which are the best preceding matches.
Example with pixels (a la Cox et al.).
CS223b

56. The Ordering Constraint
Points on the epipolar lines appear in the same order
But it is not always the case ...
This enables dynamic programming
CS223b

57. Dynamic Programming (Baker and Binford, 1981)
Find the minimum-cost path going monotonically
down and right from the top-left corner of the
graph to its bottom-right corner.
Nodes = matched feature points (e.g., edge points).
Arcs = matched intervals along the epipolar lines.
Arc cost = discrepancy between intervals.
CS223b

58. Dynamic Programming (Baker and Binford, 1981)
Find the minimum-cost path going monotonically
down and right from the top-left corner of the
graph to its bottom-right corner.
Nodes = matched feature points (e.g., edge points).
Arcs = matched intervals along the epipolar lines.
Arc cost = discrepancy between intervals.
CS223b

59. Dynamic Programming (Ohta and Kanade, 1985)
CS223b
Reprinted from “Stereo by Intra- and Intet-Scanline Search,” by Y. Ohta and T. Kanade, IEEE Trans. on Pattern Analysis and Machine
Intelligence, 7(2): (1985).  1985 IEEE.

60. Other constraints
Smoothness: disparity usually doesn’t change too quickly.
Unfortunately, this makes the problem 2D again.
Solved with a host of graph algorithms, Markov Random Fields, Belief Propagation, ….
Uniqueness constraint (each feature can at most have one match)
Occlusion and disparity are connected.
CS223b