1. Projective cameras Trifocal tensors Euclidean/projective SFM Self calibration Line geometry Purely projective cameras Je ne suis pas la la semaine prochaine. Quand peut-on rattrapper le cours? Planches : –http://www.di.ens.fr/~ponce/geomvis/lect5.pptxhttp://www.di.ens.fr/~ponce/geomvis/lect5.pptx –http://www.di.ens.fr/~ponce/geomvis/lect5.pdfhttp://www.di.ens.fr/~ponce/geomvis/lect5.pdf

2. Trinocular Epipolar Constraints These constraints are not independent!

3. Trifocal Constraints

4. All 3x3 minors must be zero! Calibrated Case

5. Trifocal Constraints All 3x3 minors must be zero! Calibrated Case Trifocal Tensor

6. Trifocal Constraints All 3x3 minors must be zero! Calibrated Case Trifocal Tensor

7. Trifocal Constraints All 3x3 minors must be zero! Calibrated Case Trifocal Tensor

8. Trifocal Constraints Uncalibrated Case Trifocal Tensor

9. Trifocal Constraints: 3 Points Pick any two lines l and l through p and p. Do it again. 23 23 T( p, p, p )=0 12 3

10. Properties of the Trifocal Tensor Estimating the Trifocal Tensor Ignore the non-linear constraints and use linear least-squares a posteriori. Impose the constraints a posteriori. For any matching epipolar lines, l G l = 0. The matrices G are singular. They satisfy 8 independent constraints in the uncalibrated case (Faugeras and Mourrain, 1995). 213 Ti 1 i

11. For any matching epipolar lines, l G l = 0. 213Ti The backprojections of the two lines do not define a line!

12. Multiple Views (Faugeras and Mourrain, 1995)

13. Two Views Epipolar Constraint

14. Three Views Trifocal Constraint

15. Four Views Quadrifocal Constraint (Triggs, 1995)

16. The Euclidean (perspective) Structure-from-Motion Problem Given m calibrated perspective images of n fixed points P j we can write Problem: estimate the m 3x4 matrices M i = [R i t i ] and the n positions P j from the mn correspondences p ij. 2mn equations in 11m+3n unknowns Overconstrained problem, that can be solved using (non-linear) least squares!

17. The Euclidean Ambiguity of Euclidean SFM If R i, t i, and P j are solutions, So are R i ’, t i ’, and P j ’, where In fact, the absolute scale cannot be recovered since: When the intrinsic and extrinsic parameters are known Euclidean ambiguity up to a similarity transformation.

18. Euclidean motion from E (Longuet-Higgins, 1981) Given F computed from n > 7 point correspondences, and its SVD F= UWV T, compute E=U diag(1,1,0) V T. There are two solutions t’ = u 3 and t’’ = -t’ to E T t=0. Define R’ = UWV T and R” = UW T V T where (It is easy to check R’ and R” are rotations.) Then [t x ’]R’ = -E and [t x ’]R” = E. Similar reasoning for t”. Four solutions. Only two of them place the reconstructed points in front of the cameras.

19. Euclidean reconstruction. Mean relative error: 3.1%

20. A different view of the fundamental matrix Projective ambiguity ! M’Q=[Id 0] MQ=[A b]. Hence: zp = [A b] P and z’p’ = [Id 0] P, with P=(x,y,z,1) T. This can be rewritten as: zp = ( A [Id 0] + [0 b] ) P = z’Ap’ + b. Or: z (b × p) = z’ (b × Ap’). Finally: p T Fp’ = 0 with F = [b x ] A.

21. Projective motion from the fundamental matrix Given F computed from n > 7 point correspondences, compute b as the solution of F T b=0 with |b| 2 =1. Note that: [a x ] 2 = aa T - |a| 2 Id for any a. Thus, if A 0 = - [b x ] F, [b x ] A 0 = - [b x ] 2 F = - bb T F + |b| 2 F = F. The general solution is M = [A b] with A = A 0 + (  b | b |  b).

22. Two-view projective reconstruction. Mean relative error: 3.0%

23. Bundle adjustment Use nonlinear least-squares to minimize:

24. Bundle adjustment. Mean relative error: 0.2%

25. From uncalibrated to calibrated cameras Weak-perspective camera: Calibrated camera: Problem: what is Q ? Note: Absolute scale cannot be recovered. The Euclidean shape (defined up to an arbitrary similitude) is recovered.

26. Reconstruction Results (Tomasi and Kanade, 1992) Reprinted from “Factoring Image Sequences into Shape and Motion,” by C. Tomasi and T. Kanade, Proc. IEEE Workshop on Visual Motion (1991).  1991 IEEE.

27. What is some parameters are known? Self calibration M= ρ K [ R t ] Q= [C d] => M (CC ) M = ρ KK TTT2 M Ω M = ρ ω with Ω = CC and ω = KK TTT * * * * If u0=v0=0, linear constraints on Ω (Triggs’97, Pollefeys’98) * 2

28. x c ξ r y c Purely projective cameras

29. x c ξ r y x c ξ Line geometry!

30. x y   = x Ç y = = = (u ; v) y – x u x £ y v The join of two points and Plücker coordinates (Euclidean version) u O v Note: u. v = 0

31. x y  The join of two points and Plücker coordinates (projective version) u O v Note: u. v = 0  = x Ç y = uvuv [ ]

32. An inner product for lines  = (u ; v) !  * = (v ; u)  = (s ; t) ! (  |  ) =  *.  = .  * = u. t + v. s   (  |  ) = 0 Note: (  |  )= 2 u. v = 0

33. line screw P5P5 the Klein quadric Interpreting Plűcker coordinates

34. p2p2 Duality x p1p1 p3p3 x. p k = 0 x * = { p | x. p = 0 }

35.  = p Æ q = ( p Ç q ) * The meet of two planes p q 

36.    = x * Æ y * = ( x Ç y ) * Line duality x*x* y*y*   x y

37. The joint of a line and a point  x p =  Ç x p = [  Ç ] x where [  Ç ] = [u £ ] v -v T 0 When is  Ç x equal to 0?

38. The meet of a line and a plane  p x =  Æ p x = [  Æ ] p where [  Æ ] = [   Ç ] When if  Æ p equal to 0?

39.    Coplanar lines and Line bundles

40. » = u 1 » 1 + u 2 » 2 + u 3 » 3 Line bundles c x 33 11 22 

41. » = u 1 » 1 + u 2 » 2 + u 3 » 3 y = u 1 y 1 + u 2 y 2 + u 3 y 3 y2y2 c r x y y1y1 33 11 22  y3y3 Line bundles

42. » = X u, where X 2 R 6 £ 3, u 2 R 3 y = Y u, where Y 2 R 4 £ 3, u 2 R 3 y2y2 c r x y y1y1 33 11 22 y3y3 Line bundles 

43. u = Y z y y = Y z y [(c Ç x) Æ r] y2y2 c r x y y1y1 33 11 22 y3y3 Line bundles  Note:

44. u = Y z y y = Y z y [(c Ç x) Æ r] y2y2 c r x y y1y1 33 11 22 y3y3 Line bundles  Note: (c Ç x) Æ r = [c x – x c ] r TT

45. u = Y z y y = Y z y [(c Ç x) Æ r] = P x when z = c y2y2 c r x y y1y1 33 11 22 y3y3 Line bundles 

46. c r x y  y ¼ P x  ¼ P * p ¼ P T   ¼ P T y Perspective projection z  p NOTE: Here u=y X=P T

47. The fundamental matrix revisited ( »   | » 2 ) = 0 y 1 T F y 2 = 0 y 1 11 22  1 ¼ P 1 T y 1  2 ¼ P 2 T y 2 y 2

48.     1  1  1  2  2  2  3  3  3  4  4  4  5  5  5  6  6  6 The trifocal tensor revisited T ( y 1, y 2, y 3 ) = 0

49. The trifocal tensor revisited D i ( » 1, » 2, » 3 ) = 0 or T i (u 1, u 2, u 3 ) = 0, for i = 1,2,3,4  1  1  1  2  2  2  3  3  3  4  4  4  5  5  5  6  6  6 δ η φ x (Ponce et al., CVPR’05)

50. c r x y  y ¼ P x  ¼ P * p ¼ P T   ¼ P T y Perspective projection z  p NOTE: Here u=y X=P T

51. П1П1 Chasles’ absolute conic: x 1 2 + x 2 2 + x 3 2 = 0, x 4 = 0. The absolute quadratic complex:  T diag(Id,0)  = | u | 2 = 0.

52. e p T = H p p T e x = H -1 p x Coordinate changes --- Metric upgrades Planes: Points: Lines: e  = p  x’ ¼ P ( H H -1 ) x H = [ X y ]

53. Perspective projection c r x x ’  c r x x ’  x’ ¼ P x  ’ ¼ P *  p’ ¼ P T  ’  ¼ P T x’ x’ ¼ P x  ¼ P *  p’ ¼ P T   ¼ P T The AQC general equation:  T  = 0, with  = X *T X * Proposition:  T  ’ ¼ û ¢ û’ Proposition : P  P T ¼  ’ p  y ’  ’ y ’  ’ Proposition : P  * P T ¼  * Triggs (1997); Pollefeys et al. (1998) e p T = H p p T e x = H -1 p x e  = p 

54. Relation between K, , and  *

55. 2480 points tracked in 196 images Non-linear, 7 images Non-linear, 20 images Non-linear, 196 images Linear, 20 images

56. Canon XL1 digital camcorder, 480 £ 720 pixel 2 (Ponce & McHenry, 2004) Projective structure from motion : Mahamud, Hebert, Omori & Ponce (2001)

57. What is a camera? (Ponce, CVPR’09) x c ξ r y x

58. x c ξ r y c

59. x c ξ r y x c ξ

60. x c ξ r y x c ξ ξ

61. x c ξ r y x ξ r y Linear family of lines x ξ x c ξ ξ ξ

62. Lines linearly dependent on 2 or 3 lines (Veblen & Young, 1910) Then go on recursively for general linear dependence © H. Havlicek, VUT

63. What a camera is Definition: A camera is a two-parameter linear family of lines – that is, a degenerate regulus, or a non-degenerate linear congruence.

64. Rank-3 families: Reguli Line fields ≡ epipolar plane images (Bolles, Baker, Marimont, 1987) Line bundles

65. Rank-4 (nondegenerate) families: Linear congruences Figures © H. Havlicek, VUT

66. x ξ y r x y r ξ Hyperbolic linear congruences Crossed-slit cameras (Zomet et al., 2003) Linear pushbroom cameras (Gupta & Hartley, 1997)

67. © E. Molzcan © Leica Hyperbolic linear congruences

68. © T. Pajdla, CTU Elliptic linear congruences Linear oblique cameras (Pajdla, 2002) Bilinear cameras (Yu & McMillan, 2004) Stereo panoramas / cyclographs (Seitz & Kim, 2002)

69. Parabolic linear congruences Pencil cameras (Yu & McMillam, 2004) Axial cameras (Sturm, 2005)

70. Plücker coordinates and the Klein quadric line screw the Klein quadric  = x Ç y = uvuv [ ] x y  Note: u. v = 0 P5P5

71. Pencils of screws and linear congruences line s P5P5 the Klein quadric Reciprocal screws: (s | t) = 0 Screw ≈ linear complex: s ≈ { ± | ( s | ± ) = 0 }

72. line s P5P5 the Klein quadric t l Pencils of screws and linear congruences Reciprocal screws: (s | t) = 0 Screw ≈ linear complex: s ≈ { ± | ( s | ± ) = 0 } Pencil of screws: l = { ¸ s + ¹ t ; ¸, ¹2 R } The carrier of l is a linear congruence

73. P5P5 e h p Reciprocal screws: (s | t) = 0 Screw ≈ linear complex: s ≈ { ± | ( s | ± ) = 0 } Pencil of screws: l = { ¸ s + ¹ t ; ¸, ¹2 R } The carrier of l is a linear congruence Pencils of screws and linear congruences

74. x ±2±2 Hyperbolic linear congruences »

75. x »1»1 p1p1 ±1±1 ±2±2 p2p2 » = (x T [ p 1 p 2 T ]x) » 1 + (x T [ p 1 p 2 T ]x) » 2 + (x T [ p 1 p 2 T ]x) » 3 + (x T [ p 1 p 2 T ]x) » 4 »2»2 »3»3 »4»4 »

76. x »1»1 p1p1 ±1±1 ±2±2 p2p2 Hyperbolic linear congruences » = (y T [ p 1 p 2 T ] y) » 1 + (y T [ p 1 q 2 T ] y) » 2 + (y T [ q 1 p 2 T ] y) » 3 + (y T [ q 1 q 2 T ] y) » 4 y = u 1 y 1 + u 2 y 2 + u 3 y 3 = Y u »2»2 »3»3 »4»4 » y

77. x »1»1 p1p1 ±1±1 ±2±2 p2p2 Hyperbolic linear congruences » = (u T [ ¼ 1 ¼ 2 T ]u) » 1 + (u T [ ¼ 1 ρ 2 T ]u) » 2 + (u T [ρ 1 ¼ 2 T ]u) » 3 + (u T [ρ 1 ρ 2 T ]u) » 4 = X û, where X 2 R 6 £ 4 and û 2 R 4 »2»2 »3»3 »4»4 » y

78. x ξ ± a2a2 p1p1 z p2p2 p a1a1 Parabolic linear congruences ± s ° T » = X û, where X 2 R 6 £ 5 and û 2 R 5

79. Elliptic linear congruences x » y » = X û, where X 2 R 6 £ 4 and û 2 R 4

80. x »1»1 y1y1 »2»2 y2y2 Epipolar geometry ( » 1 | » 2 ) = 0 or û 1 T F û 2 = 0, where F = X 1 T X 2 2 R 4 £ 4 Feldman et al. (2003): 6 £ 6 F for crossed-slit cameras Gupta & Hartley (1997): 4 £ 4 F for linear pushbroom cameras

81. Trinocular geometry D i ( » 1, » 2, » 3 ) = 0 or T i (û 1, û 2, û 3 ) = 0, for i = 1,2,3,4  1  1  1  2  2  2  3  3  3  4  4  4  5  5  5  6  6  6 δ η φ x

82. Admissible maps and intrinsic parameters (Batog, Goaoc, Ponce, CVPR’10) Optics Retina x y’ l l’ y l Because light travels along straight lines in homogeneous media, the lines associated with any camera must form a congruence of order 1 (Sturm, 1893; Benić and Gorjanc, 2006).

83. Proposition: Given two cameras with the same underlying congruence but distinct retinas, the image formed by the first camera is projectively equivalent to the image formed by the other one after some projective transformation of space. In plain English: The retinal plane matters.

84. Retina x l y Ax 2

85. Retina x l y Ax 2 Proposition: A necessary and sufficient condition for a 4x4 matrix A to be admissible—that is, induce a linear camera, is that its minimum polynomial has degree 2. Proposition: There is a bijection between admissible maps and linear cameras.

86. Intrinsic parameters Hyperbolic Parabolic Elliptic

87. Building a parabolic camera (Batog, Goaoc, Lavandier, Ponce, 2010)