1. Computer Vision No. 7: Stereo part 2

2. Today’s topics u Projective geometry u Camera Projection Matrix u F Matrix u Multi View Shape Reconstruction

3. Elementary Projective Geometry

4. Topics 1.Projective Plane 2. Projective Space 3. Affine Space 4. Euclidean Space

5. Projective Plane (2D projective space) In the Euclidean plane In the Projective plane Simply add a third coordinate of 1 at the end Homogeneous coordinate ! Scaling is NOT important. Therefore the point (0, 0, 0) is disallowed in the projective plane. : Ideal points or Points at infinity

6. Description in Projective Plane A line in the projective plane is described as… A line through the point (x, y)… A line through a point

7. Duality in Projective Plane A point and a line The line through point and If point, and are on the same line The cross point of line and If line, and are passing through the same point

8. Conics and Dual Conics For symmetric 3 x 3 matrix … All points which satisfy …. Conic Dual Conic Therefore, for the matrix All lines which satisfy ….

9. Transformation in Projective Plane In the Projective plane, by 3 x 3 non-singular matrix T A point is transformed into A line is transformed into A conic is… Similarly a dual conic is…

10. In the Euclidean space In the Projective space Homogeneous coordinate ! : Ideal points, Points at infinity A plane in the projective space is described as… A plane through a point Projective Space (3D Projective space)

11. Quadrics and Dual Quadrics For symmetric 4 x 4 matrix … All points which satisfy …. Quadric Dual Quadric Therefore, for the matrix All planes which satisfy …. Similarly a dual quadric is… A quadric is transformed by T into

12. N-Dimensional Affine Space Among transformations T in the N-d projective space Affine Transformation In N-D affine space… : point at infinity The group of the points at infinity : plane at infinity By an affine transformation a point at infinity

13. N-Dimensional Euclidean Space Among N-d affine transformations Euclidean Transformation a group of points at infinity, which satisfy By an Euclidean transformation N=3: Absolute conic N=2: Absolute point

14. 3-Dimensional Euclidean Space By an Euclidean transformation Absolute Dual Quadric The absolute dual quadric is invariant under Euclidean transformation!

15. Camera Projection

16. Topics 1.Pinhole Camera Model 2. Camera Parameters 3. Lens Distortion

17. Pinhole Camera

18. Pinhole Camera Model geometry (X, Y, Z) Image plane X Y -Z x y (x, y) Perspective projection View point (Optical center) (sX, sY, sZ)

19. Pinhole Camera Model The projected point on the image plane (z = -1) The Rotation matrix of camera pose The camera position In Projective Geometry

20. Intrinsic parameters y An ideal image on the Image plane x u v θ An actual picture u0u0 v0v0 (x, y) (u, v) the Principal point skew

21. Intrinsic parameters y An ideal image on the Image plane x u v θ An actual picture u0u0 v0v0 (x, y) (u, v) the Principal point skew f : Focal Length a : aspect ratio

22. Camera Projection Matrix Transformation of a point in the 3D world into a 2D picture… Camera Matrix (3 x 4) Camera calibration 11 parameters: rotation 3 + translation 3 + intrinsic 5 Estimate of these 11 parameters (= camera matrix)

23. Lens distortion Short focal length Long focal length

24. Removal of lens distortion

25. Lens Distortion Z.Zhang (‘00) PAMI R. Sagawa (‘05) IROS Non-parametric

26. F Matrix

27. Topics 1.Epipolar Geometry 2.E & F Matrix 3.Homography 4.Multi View Geometry

28. Correspondence Problem Given two images, we want to solve a problem…… For a point in image 1, decide which point in image 2 it corresponds to. ? plane 1 plane 2 Eye 1 Eye 2

29. Epipolar Geometry C1C1 C2C2 3D point Epipolar plane Epipole Epipolar line Baseline

30. Essential Matrix C1C1 C2C2 Essential matrix : E

31. Essential & Fundamental Matrix Image planes Pictures Fundamental matrix : F Image 1 Image 2

32. Fundamental Matrix image 1 image 2 constraint Depends on the camera configuration!!

33. Fundamental Matrix are on the same plane! Three vectors

34. Epipolar Constraint

36. Property of F Matrix at most Rank 2 There are some points which satisfy

37. Null Space of F Matrix (epipole) for All epipolar lines in image 2 pass through the point

38. Null Space of F Matrix (epipole) Similarly,for : Epipoles All epipolar lines in image 1 pass through the point

39. Epipoles Epipoles: intersections of baseline with image planes projection of the optical center in another image the vanishing points of camera motion direction C1C1 C2C2 e1e1 e2e2

40. Examples

43. Computing F Matrix (8-point algorithm)

44. Computing F Matrix Epipolar pencil by linear solution (due to noise and error)

45. Computing F Matrix Singular value decomposition (SVD) Without noise, σ 3 must be 0 modification

46. Computing F Matrix

47. Homography P In Camera 1 Camera 1 Camera 2 The distance between camera 1 and the plane In Camera 2

48. Homography and F Matrix P Camera 1 Camera 2

49. Planer homography mosaicing

50. camera Image plane’

51. Image rectification Camera 1Camera 2

52. Rectification Fusiello (MVA 2000) Pollefeys

53. Constraints of 3 images LLine – line – line 3 x 3 matrix

54. Constraints of 3 images L Line – line – line Tensor description L Point – line – line

55. Constraints of 3 images Point – point – line L Point – point – point

56. 2-Views Geometry X M : (3 x 2) x (4 + 2) matrix

57. 2-Views Geometry X M : (3 x 2) x (4 + 2) matrix In tensor form

58. 3-Views Geometry X M : (3 x 3) x (4 + 3) matrix

59. 4-Views Geometry X

60. Camera Calibration

61. Topics 1.3D Ref. 2.2D Ref. 3. 1D Ref. 4. Self Calibration (Kruppa equations)

62. The meaning of camera calibration Absolute Dual Quadric Camera Calibration = to estimate the image of the absolute quadric

63. Camera Calibration (3D Ref.) known

64. Homography （ 2D-2D mapping ） (x, y) (u, v) Homography (3 x 3) the origin

65. Camera Calibration (2D Ref.) Without loss of generality… Z.Zhang (‘00) PAMI

66. Camera Calibration (2D Ref.) known

67. Camera Calibration (2D Ref. Results)

68. Camera Calibration (1D Ref.) Z.Zhang (‘04) PAMI On the line in 3D :const.:fix

69. Camera Calibration (1D Ref. Results)

70. Self Calibration In most cases, there are NOT any reference objects… be able to calibrate the camera by unknown scenes

71. Homography at infinity P Camera 1 Camera 2 Considering P on the infinity plane … and on the absolute quadric… In camera 2 coordinate system

72. The image of the absolute quadric P Camera 1 Camera 2 The image of the absolute quadric at the infinity plane

73. The Kruppa equations P Camera 1 Camera 2 Given the F Matrix…

74. References u Multiple View Geometry Hartley and Zisserman, Cambridge u The Geometry of Multiple Images Faugeras and Luong, MIT u Three-Dimensional Computer Vision Faugeras, MIT u Algebraic Projective Geometry Semple and Kneebone, Oxford

75. Multiple View Shape Reconstruction

76. Topics Basic Principle Surface Approach  Multi-baseline stereo Volumetric Approach  Volume intersection method Combinatorial Approach  Photo hull Extended Algorithm Level Set Approach Graph-cut Approach

77. Basic Principle #1 Surface Approach

78. u Try to reconstruct a surface of an object –Find depth d that minimizes some matching cost function F (d) defined on N views Camera 1 Camera 2 Camera N

79. Matching Cost Function (b) SSSD (sum of sum. of squared difference) (a) SSAD (sum. of sum of absolute difference) (c) SNCC (sum of normalized cross correlation) Camera 1 (reference) Camera 2 Camera N Epipolar line

80. Why Multiple Views? u Binocular stereo is sometimes ambiguous e.x. F(d) = SAD (sum. of absolute difference) d Camera 1Camera 2 d Which is the right match? b

81. Multi-baseline Stereo T. Kanade, M. Okutomi, and T. Nakahara, "A Multiple-Baseline Stereo Method," Proc. ARPA Image Understanding Workshop, pp. 409-426, 1992. Camera 1 (reference) Camera 2 d Camera 3Camera N d SAD 2 d SSAD d SAD 3 d SAD N Unique minima

82. Video-rate Stereo Vision u Hardware implementation is easy S. Kimura, T. Kanade, H. Kano, A. Yoshida, E. Kawamura, K. Oda, “CMU video-rate stereo machine", Proc. of Mobile Mapping Symposium, 1995 Image capture Apply LoG filter SSAD computation Disparity computation Block matching LUT instead of rectification SAD computation

83. Real-time Robot Vision 9 cameras stereo

84. Real-time Robot Vision u Model-based 3D Object Tracking

85. Virtualized Reality ™ u 51 video cameras surrounding the scene –Recorded to VCR –Off-line 3D modeling T. Kanade, P. Rander, P. J. Narayanan, “Virtualized Reality: Constructing Virtual Worlds from Real Scenes”, IEEE MultiMedia, vol.4, no.1, pp.34-47, 1997 Large scene from overlapped dense stereo images Input images 3D video

86. Basic Principles #2 Volumetric Approach

87. u Try to reconstruct a volume that contains an object –Detect silhouette in each image –Backproject each silhouette –Intersect backprojected volumes Binary Images

88. Volume Intersection u Reconstruction contains the true scene –But is generally not the same

89. Visual Hull u Minimum convex hull of an object ⇒ A volume obtained from infinite number of surrounding cameras Visual hull True shape Reconstructed volume Concave area is never recovered Bound of the reconstruction

90. Voxel Algorithm for Volume Intersection u Find voxel that is projected inside silhouette in all the images –O(NM 3 ), for N images, M 3 voxels –Don ’ t have to search 2 M 3 possible scenes! Octree representation

91. Acceleration via Octree u Find voxel that is projected inside silhouette in all the images –Multi-resolution approach –Voxels that project to only background pixels in any image are carved –Voxels that project to only foreground pixels in all images remain –Ambiguous voxels are subdivided Resolution 1

92. Acceleration via Octree u Find voxel that is projected inside silhouette in all the images –Multi-resolution approach –Voxels that project to only background pixels in any image are carved –Voxels that project to only foreground pixels in all images remain –Ambiguous voxels are subdivided Resolution 2

93. Acceleration via Octree u Find voxel that is projected inside silhouette in all the images –Multi-resolution approach –Voxels that project to only background pixels in any image are carved –Voxels that project to only foreground pixels in all images remain –Ambiguous voxels are subdivided Resolution 3

94. Results u Reconstruction from 4 synthetic images Silhouette images 16x16x16 voxel space 32x32x32 voxel space64x64x64 voxel space

95. How do we get silhouette images? u Dependent on the purpose of the application –Interactive Foreground Separation »Building static 3d model from small number of images –Statistical Background Subtraction »Motion capture system in controlled environment from large number of image sequence (multiple video streams)

96. Interactive Foreground Separation u Use of an interactive segmentation program Y. Li, J. Sun, C.K. Tang, H.Y. Shum, “Lazy snapping,” ACM Trans. on Graphics Vol.23,No. 3, pp.303—308, 2004. Foreground regionBackground region SeparationWrong segmentation Background region Separation See “Example of Graph-cut” in the lecture 6

97. Building Static 3d Model u Volume intersection from 8 images –No concavities

98. Brightness component Chromaticity component Statistical Background Subtraction Background imageImage of interest ifthen pixel Ii is foreground else if then pixel Ii is shadow else pixel Ii is shadow

99. Choice of Threshold u Learn statistics of brightness and chromaticity component for each pixel Brightness distributionChromaticity distribution P % P is around 99%

100. Motion Capture System in Controlled Environment Input image Foreground & shadow Silhouette image

101. Basic Principles #3 Combinatorial Approach

102. 1. Start with an initial volume uArbitrary rectangular shape uResult of volume intersection 2. The volume is iteratively carved to be getting closer to the true shape uConsidering photo-consictency u2 approaches: Voxel coloring and Space-carving Carve an inconsistent surface voxel

103. Photo-consistency u Color variance of the imaged voxel among the cameras If color variance > threshold the voxel is carved If color variance < threshold the voxel is preserved Voxel of interest True surface

104. Voxel Coloring Approach Input Images (Calibrated) Goal: Assign RGBA values to voxels in V photo-consistent with images u Discrete formulation

105. 1. Choose voxel 2. Project and correlate 3. Color if photo-consistent Visibility Problem In what order do we have to choose a voxel? u In which images is each voxel visible?

106. The Global Visibility Problem u Inverse Visibility u known images Unknown Scene u Which points are visible in which images? Known Scene u Forward Visibility u Unknown images

107. Layers Solution: Depth Ordering SceneTraversal Condition: voxel space and group of cameras must be completely separated voxel space and group of cameras must be completely separated u Visit occluders first

108. Space-carving Algorithm u General case Image 1 Image N …... –Initialize to a volume V containing the true scene –Repeat until convergence –Choose a voxel on the current surface –Carve if not photo-consistent –Project to visible input images

109. Output of Space-carving: Photo hull u The Photo Hull is the UNION of all photo-consistent scenes in V –Tightest possible bound on the true scene True Scene V Photo Hull V

110. u Algorithm finds a surface consistent with the input images, but not necessarily the actual surface Typical Error in Photo hull actual surface reconstructed surface Cusping artifact Floating voxel artifact

111. Space-carving Implementation u Multi-pass plane sweep –Efficient, easy to implement Sweep in –z direction Sweep in -x directionSweep in -y direction Repeat

112. Results Details are recovered Volume intersection Space carving with textures Excessively carved Input image Using 16 images 300k polygons

113. Extended Algorithm #1 Level Set Approach

114. u Define problem in 1 higher dimension –An additional dimension, time t, is added φ(p(t),t). [Osher98] »P(t) is a point in the space »Φ(p(t),t) is a signed distance function –Surface, zero level set φ=0, moves along its normal direction –Front propagation is guided by Partial Differential Equation (PDE). –Cross-correlation is used as a constraint in PDE. [Faugeras98] True shape

115. How to move the level set surface u Define a velocity field, F, that specifies surface points move in time –In the case of stereo, F is a function of the matching cost on p(t) in multi-baseline stereo –F decreases to 0 around the true surface u Build an initial value for the level set function, φ(p(0),0), based on the initial surface u Adjust φ over time –Surface at time t is defined by φ(p(t),t)=0

116. Level Set Formulation u Constraint: level set value of a point p(t) on the surface must always 0 u By the chain rule u Since F supplies the speed in the outward normal direction u Hence evaluation equation for φ is Essentially a gradient-descent local optimization problem

117. Front Propagation by PDE u Front Propagation can be efficiently calculated in level set framework –Fast marching method –Narrow band method

118. Results u Can handle topology change

119. Extended Algorithm #2 Graph-cut Approach

120. Goal Calibrated images of Lambertian scene 3D model of scene

121. Volumetric Graph-cuts u Energy function to minimize Outer surface (Visual hull) Inner surface (at constant offset from outer surface) Volume between outer surface and S (to avoid over-smoothing) Reconstructed surface (S) Integral of photo-consistency over S (S should be photo-consistent) Photo-consistency

122. Designing a Graph 1. Assign a node for each voxel 2. Connect neighboring nodes –Weight is set as ρ(x) 3. Connect outer nodes to source –Weight is infinite 4. Connect inner nodes to sink –Weight is infinite Outer surface Inner surface Sink Source

123. Graph-cut algorithm minimizes the energy Designing a Graph 1. Connect all the nodes to source –Weight is λ Outer surface Inner surface Sink Source A cut that separates source and sink is equal to Cut

124. Results Volume intersection Volumetric Graph-cut

125. Summary u Multi-baseline stereo for accurate depth map –Good for real-time applications, dense depth map is obtained directly u Voxel-based reconstruction –Volume intersection for conservative estimate –2 background subtraction methods for different applications –Concavities can be recovered by considering Photo-consistency u Extended algorithm –2 approaches to add smoothness constraint –Level set approach solves local optimization problem »can handle topology change –Graph-cut approach solves global optimization problem »cannot handle topology change

126. u Volume intersection –Martin & Aggarwal, “ Volumetric description of objects from multiple views ”, Trans. Pattern Analysis and Machine Intelligence, 5(2), 1991, pp. 150-158. –Szeliski, “ Rapid Octree Construction from Image Sequences ”, Computer Vision, Graphics, and Image Processing: Image Understanding, 58(1), 1993, pp. 23-32. u Voxel coloring and Space-carving –Seitz & Dyer, “ Photorealistic Scene Reconstruction by Voxel Coloring ”, Proc. Computer Vision and Pattern Recognition (CVPR), 1997, pp. 1067-1073. –Kutulakos & Seitz, “ A Theory of Shape by Space Carving ”, Proc. ICCV, 1998, pp. 307-314. u Extended algorithm –Faugeras & Keriven, “ Variational principles, surface evolution, PDE's, level set methods and the stereo problem", IEEE Trans. on Image Processing, 7(3), 1998, pp. 336-344. –G. Vogiatzis and P. H. S. Torr and R. Cipolla, “Multi-View Stereo via Volumetric Graph-Cuts”, Proc. CVPR, 2005, pp.391-398. References

127. Next Week u 6/7 Motion understanding –Prof. Ikeuchi