1. 3D Vision and Graphics

2. The Geometry of Multiple Views Motion Field Stereo Epipolar Geometry The Essential Matrix The Fundamental Matrix Structure from Motion

3. Multiple View/Motion Research Motion Research Motion Segmentation Tracking Structure From Motion Infinitesimal SFM Multiview Methods Orthographic/AffinePerspectiveUncalibratedStereo

4. The Motion Field

5. Motion “When objects move at equal speed, those more remote seem to move more slowly.” - Euclid, 300 BC

6. The Motion Field Where in the image did a point move? Down and left

7. Motion Field Equation v x, v y : Components of image motion T: Components of 3-D linear motion  Angular velocity vector (x,y): Image point coordinates Z: depth f: focal length

8. Pure Translation 

9. Forward Translation & Focus of Expansion [Gibson, 1950]

10. Pure Rotation: T=0 Independent of T x T y T z Independent of Z Only function of (x,y), f and 

11. Pure Rotation: Motion Field on Sphere

12. Binocular Stereo Binocular Stereo Some slides on stereo adapted from Kristian Kirk

13. Traditional Application: Mobile Robot Navigation The Stanford Cart, H. Moravec, 1979. The INRIA Mobile Robot, 1990. Courtesy O. Faugeras and H. Moravec.

14. What is binocular stereo vision?  A way of getting depth (3-D) information about a scene from two 2-D views (images) of the scene

15. What is binocular stereo vision?  A way of getting depth (3-D) information about a scene from two 2-D views (images) of the scene  Used by humans and animals

16. What is binocular stereo vision?  A way of getting depth (3-D) information about a scene from two 2-D views (images) of the scene  Used by humans and animals  Computational stereo vision  Programming machines to do stereo vision  Studied extensively in the past 25 years  Difficult; still being researched

17. Fundamentals of Stereo Vision  A camera model:  Models how 3-D scene points are transformed into 2-D image points  The pinhole camera: a simple linear model for perspective projection

18. Fundamentals of Stereo Vision  The goal of stereo analysis:  The inverse process: From 2-D image coordinates to 3-D scene coordinates  Requires images from at least two views

19. Fundamentals of Stereo Vision  3-D reconstruction

20. Fundamentals of Stereo Vision  3-D reconstruction

21. Fundamentals of Stereo Vision  3-D reconstruction

22. Fundamentals of Stereo Vision  3-D reconstruction

23. Fundamentals of Stereo Vision  3-D reconstruction

24. Fundamentals of Stereo Vision  3-D reconstruction

25. Fundamentals of Stereo Vision  3-D reconstruction

26. Fundamentals of Stereo Vision  3-D reconstruction

27. Prerequisites  Camera model parameters must be known:  External parameters:  Positions, orientations  Internal parameters:  Focal length, image center, distortion, etc..

28. Prerequisites  Camera calibration

29. Two subproblems  Matching  Finding corresponding elements in the two images  Reconstruction  Establishing 3-D coordinates from the 2-D image correspondences found during matching

30. Two Subproblems  Matching (hardest)  Finding corresponding elements in the two images  Reconstruction  Establishing 3-D coordinates from the 2-D image correspondences found during matching

31.  Which image entities should be matched?  Two main approaches  Pixel/area-based (lower-level)  Feature-based (higher-level) The Matching Problem

32. Matching Challenges  Scene elements do not always look the same in the two images  Camera-related problems  Image noise, differing gain, contrast, etc..  Viewpoint-related problems:  Perspective distortions  Occlusions  Specular reflections

33. Choice of Camera Setup  Baseline  distance between cameras (focal points)  Trade-off  Small baseline: Matching easier  Large baseline: Depth precision better

34. Matching Clues  Correspondance search is a 1-D problem  Matching point must lie on a line

35. Matching Clues  Epipolar geometry

36. Matching Clues  Epipolar geometry

37. Rectification  Simplifies the correspondance search  Makes all epipolar lines parallel and coincident  Corresponds to parallel camera configuration

38. Rectification All epipolar lines are parallel in the rectified image plane.

39. Reconstruction from Rectified Images Disparity: d=u’-u.Depth: z = -B/d.

40. Goal: Disparity Map  Disparity:  The horizontal displacement between corresponding points  Closely related to scene depth

41. Epipolar geometry example

42. image I(x,y) image I´(x´,y´) Disparity map D(x,y) (x´,y´)=(x+D(x,y),y)

43. Human Stereopsis: Binocular Fusion How are the correspondences established? Julesz (1971): Is the mechanism for binocular fusion a monocular process or a binocular one?? There is anecdotal evidence for the latter (camouflage). Random dot stereograms provide an objective answer

44. A Cooperative Model (Marr and Poggio, 1976) Excitory connections: continuity Inhibitory connections: uniqueness Iterate: C = S C - w S C + C. ei0 Reprinted from Vision: A Computational Investigation into the Human Representation and Processing of Visual Information by David Marr.  1982 by David Marr. Reprinted by permission of Henry Holt and Company, LLC.

45. More Matching Heuristics  Always valid:  (Epipolar line)  Uniqueness  Minimum/maximum disparity  Sometimes valid:  Ordering  Local continuity (smoothness)

46. Area-based Matching  Finding pixel-to-pixel correspondences  For each pixel in the left image, search for the most similar pixel in the right image

47. Area-based Matching  Finding pixel-to-pixel correspondences  For each pixel in the left image, search for the most similar pixel in the right image  Using neighbourhood windows

48. Area-based Matching  Similarity measures for two windows  SAD (sum of absolute differences)  SSD (sum of squared differences)  CC (cross-correlation)  …

49. Area-based Matching  Choice of window size  Factors to considers:  Ambiguity  Noise sensitivity  Sensitivity towards viewpoint-related distortions  Expected object sizes  Frequency of depth jumps

50. Correlation Methods (1970--) Slide the window along the epipolar line until w.w’ is maximized.

51. Correlation Methods: Foreshortening Problems Solution: add a second pass using disparity estimates to warp the correlation windows, e.g. Devernay and Faugeras (1994). Reprinted from “Computing Differential Properties of 3D Shapes from Stereopsis without 3D Models,” by F. Devernay and O. Faugeras, Proc. IEEE Conf. on Computer Vision and Pattern Recognition (1994).  1994 IEEE.

52. Dynamic Programming (Baker and Binford, 1981) Find the minimum-cost path going monotonically down and right from the top-left corner of the graph to its bottom-right corner. Nodes = matched feature points (e.g., edge points). Arcs = matched intervals along the epipolar lines. Arc cost = discrepancy between intervals.

53. Dynamic Programming (Ohta and Kanade, 1985) Reprinted from “Stereo by Intra- and Intet-Scanline Search,” by Y. Ohta and T. Kanade, IEEE Trans. on Pattern Analysis and Machine Intelligence, 7(2):139-154 (1985).  1985 IEEE.

54. Hierarchical stereo matching Downsampling (Gaussian pyramid) Disparity propagation Allows faster computation Deals with large disparity ranges ( Falkenhagen´97;Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)

55. Disparity map image I(x,y) image I´(x´,y´) Disparity map D(x,y) (x´,y´)=(x+D(x,y),y)

56. Example: reconstruct image from neighboring images

57. Three or More Viewpoints  More matching information  Additional epipolar constraints  More confident matches

58. I1I2 I10 Reprinted from “A Multiple-Baseline Stereo System,” by M. Okutami and T. Kanade, IEEE Trans. on Pattern Analysis and Machine Intelligence, 15(4):353-363 (1993). \copyright 1993 IEEE.

59. Epipolar Geometry

60. Triangulation Nalwa Fig. 7.2

61. Epipolar Constraint Potential matches for p have to lie on the corresponding epipolar line l’. Potential matches for p’ have to lie on the corresponding epipolar line l.

62. Epipolar Geometry Epipolar Plane Epipoles Epipolar Lines Baseline

63. Epipolar Constraint: Calibrated Case Essential Matrix (Longuet-Higgins, 1981)

64. Properties of the Essential Matrix E p’ is the epipolar line associated with p’. E p is the epipolar line associated with p. E e’=0 and E e=0. E is singular. E has two equal non-zero singular values (Huang and Faugeras, 1989). T T

65. Perspective Projection  8-point algorithm  Uncalibrated cameras

66. Epipolar Constraint: Uncalibrated Case Fundamental Matrix (Faugeras and Luong, 1992)

67. Properties of the Fundamental Matrix F p’ is the epipolar line associated with p’. F p is the epipolar line associated with p. F e’=0 and F e=0. F is singular. T T

68. E vs. F The Essential Matrix E:  Encodes information on the extrinsic parameters only  Has rank 2 since R is full rank and [T x ] is skew & rank 2  Its two non-zero singular values are equal  5 degrees of freedom The Fundamental Matrix F:  Encodes information on both the intrinsic and extrinsic parameters  Also has rank 2 since E is rank 2  7 degrees of freedom

69. The Eight-Point Algorithm (Longuet-Higgins, 1981) | F | =1. Minimize: under the constraint 2

70. Non-Linear Least-Squares Approach (Luong et al., 1993) Minimize with respect to the coefficients of F.

71. Recovering Motion Recall epipolar constraint Where Solve for R & t

72. Rectification Given a pair of images, transform both images so that epipolar lines are scan lines. Estimate a Homography to apply to each image, given at least four corresponding points

73. Structure from Motion: Uncalibrated cameras and projective amibguity

74. Projective Ambiguity  We see that the same set of corresponding image points could come from two different sets of real world points and therefore both sets satisfy:

75. Factorization Method: Orthographic Cameras C. Tomasi, T. Kanade, Shape and Motion from Image Streams: A Factorization Method, IJCV, 9(2), 1992, 137-154.

76. Factorization Notation N: Number of frames n: Number of points P j : j th point in 3-D R i, T i : Rotation & Translation of Camera i (x i,j, y i,j ): image of j th point measured in the i th frame.

77. Origin of 3D Points at 3-D Centroid CentroidCentered Data Points

78. Origin of image at image Centroid Centered Image Points Centriod of image points is projection of 3D centroid

79. Data Matrix are the i,j-th element of the N by n data matrices X, Y 2N by n registered data matrix W3 Rank Theorem: Without noise, the rank of W is less than or equal to 3.

80. The Rank Theorem: Factorized The registered measurement matrix can be expressed in a matrix form: represents the camera rotation is the shape matrix

81. Factoring Given a data matrix containing measured feature points, it can be factored using singular value decomposition as: Where D: n by n diagonal matrix, non-negative entries, called singular values U: 2N by n with orthogonal columns V T : n by n with orthogonal columns Without noise,  i =0, i>3 With noise, set  i =0, i>3

82. Factoring: After setting  i =0, i>3 Where D’: 3 by 3 diagonal matrix, non-negative entries U: 2N by 3 with orthogonal columns V T : 3 by n with orthogonal columns

83. Ambiguity  True for any A.  So, find an A such that rows of RA are unit length and pairs corresponding to same image are orthogonal. Estimated camera orientation Estimated 3-D structure

84. Four of 150 input images

85. Tracked Corner Features

86. 3-D Reconstruction

87. Building

88. Reconstruction Reconstruction after Triangulation and Texture Mapping