1. 1 todo Fix up motivation slides (?) Duality Single effective viewpoint constraint Demonstrate equivalence (?) Explain IAC Linearization of projection Include reconstruction ambiguity animations, house Dimension argument, how to make intuitive Essential harmonic transform SFM motion section Self-calibration Multiple view geometry Stereo rectification Differential case Estimation algorithms References slide

2. Short Course on Omnidirectional Vision Short Course on Omnidirectional Vision International Conference on Computer Vision October 10th, 2003 Dr. Christopher Geyer Univeristy of California, Berkeley Prof. Tomáš Pajdla Center for Machine Perception Czech Technical University Prof. Kostas Daniilidis GRASP Lab University of Pennsylvania

3. 3 Outline A tour of omnidirectional systems Intro:A tour of omnidirectional systems Structure-from-motion with parabolic mirrors Part 1:Christopher: Structure-from-motion with parabolic mirrors 10 minute break Images as homogeneous spaces Part 3: Kostas: Images as homogeneous spaces Panoramic and other non-central cameras Part 4: Tomáš: Panoramic and other non-central cameras Conclusion

4. 4 Introduction In this course we will take a detailed look at omni- directional sensors for computer vision. Omnidirectional sensors come in many varieties, but by definition must have a wide field-of-view. ~180º FOV wide FOV dioptric cameras (e.g. fisheye) ~360º FOV polydioptric cameras (e.g. multiple overlapping cameras) >180º FOV catadioptric cameras (e.g. cameras and mirror systems)

5. 5 Introduction Q: Why are perspective systems insufficient and why is field of view important? A: Perspective systems are one imaging modality of many, we are interested in sensors better suited to specific tasks. Sensor modality should enter into design of computer vision systems For example, perhaps for flight wide field-of-view sensors are appropriate, and in general useful for mobile robots.

6. 6 From the Page of Omnidirectional Vision http://www.cis.upenn.edu/~kostas/omni.html Which one?

7. 7 (Poly-)Dioptric solutions Pros: - High resolution per viewing angle Cons: - Bandwidth - Multiple cameras One to two fish-eye cameras or many synchornized cameras

8. 8 (Poly-)Dioptric solutions Homebrewed polydioptric cameras are cheaper, but require calibrating and synchronizing; commercial designs tend to be expensive One to two fish-eye cameras or many synchornized cameras

9. 9 Catadioptric solutions Usually single camera combined with convex mirror Cons: - Blindspot - Low resolution Pros: - Single image

10. 10 Q: What kind of sensor should one use? A: Depends on your application. 1.If you are primarily concerned with: – resolution– surveillance (coverage) and can afford the bandwidth & expense, you might stick with polydioptric solutions 2.If you are concerned with – bandwidth –servoing, SFM investigate catadioptric or single wide FOV dioptric solutions Confused? Confused?

11. 11 Other myths and hesitations… Myth: Catadioptric images are by necessity highly distorted. Truth: Actually no; parabolic mirrors induce no distortion (perpendicular to the viewing direction). Myth: Omnidirectional cameras are more complicated than perspective cameras, and harder to do SFM with. Truth: Actually no; parabolic mirrors are easy to model, calibrate and do SFM with. Truth: Omnidirectional systems have lower resolution Tradeoff: Balance resolution and field of view for your needs

12. 12 Goals for this part of the course: Demystifying catadioptric cameras Simplify: Catadioptric projections can be described by simple, intuitive models Revelations: Modeling catadioptric projections can actually give us insight into perspective cameras SFM: To give a framework for studying structure- from-motion in parabolic cameras

13. Modeling central catadioptric cameras Part I: Modeling central catadioptric cameras

14. 14 Outline of Part I 1.Properties of arbitrary camera projections, caustics 2.The fixed viewpoint constraint 3.The central catadioptric projections 4.Models of their projections 5.A “unifying model” of central catadioptric projection 6.Consequences of the model 7.Application

15. 15 When is a catadioptric camera equivalent – up to distortion – to a perspective one? Suppose we are given a catadioptric image h For what kinds of mirrors can the image be warped by h into a perspective image? h

16. 16 Review: The projection induced by a camera f The projection induced by a camera is the function from space to the image plane, e.g.

17. 17 Review: The projection induced by a camera The least restrictive assumption that can be made about any camera model is that the inverse image of a point is a line in space The projection induced by a camera is the function from space to the image plane, e.g. f -1 (p)

18. 18 Review: The projection induced by a camera For many cameras, all such lines do not necessarily intersect in a single point

19. 19 Some optics: Caustics Their envelope is called a (dia-)caustic and represents a locus of viewpoints For many cameras, all such lines do not necessarily intersect in a single point

20. 20 Review: Central projections If all the lines intersect in a single point, then the system has a single effective viewpoint and it is a central projection

21. 21 Review: Central projections If all the lines intersect in a single point, then the system has a single effective viewpoint and it is a central projection If a central projection takes any line in space to a line in the plane, then it must be a perspective projection

22. 22 When is a catadioptric camera equivalent – up to distortion – to a perspective one? g If the projection induced by a catadioptric camera is at most a scene independent distortion of a perspective projection, then it must at least be a central projection

23. 23 When is a catadioptric camera equivalent – up to distortion – to a perspective one? If the projection induced by a catadioptric camera is at most a scene independent distortion of a perspective projection, then it must at least be a central projection The lines in space along which the image is constant intersect in a single effective viewpoint

24. 24 When is a catadioptric camera equivalent – up to distortion – to a perspective one? Question: Which combinations of mirrors and cameras give rise to a system with a single effective viewpoint?

25. 25 Central catadioptric solutions Theorem [Simon Baker & Shree Nayar, CVPR 1998]: A catadioptic camera has a single effective viewpoint if and only if the mirror’s cross-section is a conic section parabolic mirror & orthographic camera hyperbolic mirror & perspective camera elliptic mirror & perspective camera

26. 26 The fixed viewpoint constraint [Baker] x y y = f (x)  Suppose that the height of the mirror at x is f (x) And the single effective viewpoint lies a distance  from the camera focus

27. 27 The fixed viewpoint constraint The condition that the a ray emanating from the focus is reflected in a direction incident with the mirror focus can be described by an ODE

28. 28 The solutions to this ODE can be shown to be restricted to conic sections, e.g., The fixed viewpoint constraint

29. 29 Modeling a parabolic projection space point image point

30. 30 Modeling a hyperbolic projection image point space point

31. 31 Modeling an elliptic projection space point image point

32. 32 Questions about catadioptric projections Q: What are the properties of the projections induced by these types of sensors? Q: How can we extend a theory of structure-from-motion and self-calibration for uncalibrated catadioptric cameras? Note: there is no difference for calibrated catadioptric cameras, since they can be warped to calibrated perspective images Q: Are there simplified models for all catadioptric projections?

33. 33 Abstracting catadioptric projections In each case the projection to one surface (the mirror) followed by a projection to another surface (the image plane).

34. 34 Abstracting catadioptric projections In other words they can be written as the composition of two functions –f  is a non-linear function (projection to a quadric) and g is a linear function (projection to a plane)

35. 35 Abstracting catadioptric projections (projective linear) (non-linear)

36. 36 Switcharoo... Can we commute the decomposition such that the non- linear projection becomes independent of the eccentricity? (linear) (non-linear & parameter dependent) (linear) (non-linear but parameter independent)

37. 37 Decomposing catadioptric projections The general catadioptric projection can be written: The result can be written in homogeneous coordinates: (linear projective transformation) (non-linear but parameter independent) (homogeneous coordinates)

38. 38 Alternative decomposition f ´ centrally projects to the sphere; it yields a homogeneous point whose fourth coordinate is the distance of the space point f ´f ´

39. 39 Alternative decomposition g´ centrally projects to the image plane from a point on the axis of the sphere, the height of the point is determined by the eccentricity g´g´  

40. 40 Consequences of this model (1 of 5) elliptic mirror parabolic mirror stereographic projection hyperbolic mirrorplanar mirror central/perspective projection 1.Central catadioptric projections and perspective projections are represented in one framework

41. 41 Consequences of this model (2 of 5) (c) Since stereographic projection sends great circles to circles in the image, the parabolic projection of a line is a circle 2. (a) The projection of a line in space to the sphere is a great circle; (b) The central projection of a great circle to the image plane is a conic section

42. 42 Consequences of this model (3 of 5) 3. (a) The Jacobian from the viewing sphere to the image plane is easily calculated (b) The Jacobian for the para- bolic/stereographic projection is proportional to a rotation; parabolic projection is locally distortionless, i.e. conformal

43. 43 Consequences of this model (4 of 5) 4. (a) Height function is not one-to-one (b) Satisfies : reciprocal eccentricities map to the same height (c) Elliptical and hyperbolic mirrors are indistinguishable from the projections they induce

44. 44 Recall properties of perspective projection: Domain: (projective space) Range: (the projective plane) a.Antipodal points have same image b.Equator projects to line at infinity Consequences of this model (5 of 5)

45. 45 Recall properties of perspective projection: Domain: (projective space) Range: (the projective plane) a.Antipodal points have same image b.Equator projects to line at infinity Consequences of this model (5 of 5) 5.Parabolic projection: Domain: (exclude plane at  ) Range: (ext’d real plane) a.Antipodal points have inverted imgs b.Equator projects to circle propor- tional to focal length

46. 46 Modeling central catadiopric cameras 1.Unifying model of central catadioptric cameras 2.Line images are conics 3.Conformality of stereographic projection 4.Indistinguishability of elliptic and hyperbolic projections 5.Inadequacy of projective plane for catadioptric systems

47. End of Part I Questions?

48. Focus on Parabolic Mirrors Part II: Focus on Parabolic Mirrors

49. 49 Outline of Part II 1.Point, circle and line image representation 2.The image of the absolute conic 3.Lorentz transformations and their conformality 4.Infinitessimal generators of Lorentz transformations 5.Comparison of Lorentz transformations and rotations 6.Complex representation & estimation 7.Linearization of the parabolic projection

50. 50 Projective geometry: A framework for perspective imaging X p Linear projection formula because of the use of homogeneous coordinates If and then where and K is the calibration matrix z y x (R,t)

51. 51 Projective geometry: A framework for perspective imaging p Linear projection formula because of the use of homogeneous coordinates If and then where and K is the calibration matrix z y x Where, for example, the line between two points can be represented by the cross product of two points: Lines lie in the dual space q

52. 52 Projective geometry: A framework for perspective imaging Some things maybe we take for granted: Representation of lines & points Condition that a line coincide with a point Construction of the point coinciding with two lines Dually, construction of the line between two points Conditions for the coincidence of three lines Dually, conditions for the collinearity of three points Homographies and the invariance of the cross-ratio Absolute conic and all that jazz

53. 53 Over 2 decades this framework has been used to derive: Multiview geometry: multilinear constraints in multiple views E.g., fundamental matrices in two views Theories of self-calibration Simplifications for special motions: homographies, etc. Q: How can we extend these results to catadioptric cameras? A: Like the perspective theory, start with a framework for the representation of features Projective geometry: A framework for perspective imaging

54. 54 But what about non-linearity? It seems an obstacle to this vision is the non-linearity of the projection equation: Recall though that the perspective projection is also non-linear: (projection mapping induced by a parabolic catadioptric camera) homogenization

55. 55 Representation of circles Start with a circle in the image plane; this sphere is not necessarily calibrated

56. 56 Representation of circles The inverse stereographic projection of a circle is a circle

57. 57 Representation of circles Through this circle there passes a unique plane; all such planes are in 1-to-1 correspondence with circles in the image plane

58. 58 Representation of circles This plane is in 1-to-1 correspondence with its pole: the vertex of the cone tangent to the sphere at the circle

59. 59 Representation of circles This plane is in 1-to-1 correspondence with its pole: the vertex of the cone tangent to the sphere at the circle

60. 60 Representation of circles The circle’s center is collinear with the representation and the north pole, the radius varies with position along the line

61. 61 Representation of circles Given the position in space, determine the circle center and radius Easy: Solve for u, v and r

62. 62 Representation of circles Three cases: a. inside sphere b. on sphere c. outside sphere real locus, r > 0 zero radius, r = 0 imaginary locus, r imaginary

63. 63 Partition of feature space Point features and circles have point representations in the same space; recall in projective plane, dual space req’d

64. 64 Representation of image points So now we have a repre- sentation of image points For if then

65. 65 Representation of image points Recall that the sphere is the locus of points which satisfy the equation: In projective space this is the set of points lying on the quadratic surface given by a quadratic form

66. 66 Partition of feature space

67. 67 Coincidence condition What is the condition that p lies on a circle of radius r and center (x,y)? r (u,v)(u,v) p = (x,y) ax + by + c = 0 or

68. 68 Coincidence condition We want: with some algebra one finds: r (u,v)(u,v) p = (x,y)

69. 69 Angle of intersection Hence, circles orthogonal iff

70. 70 Meets and joins Projective plane: Parabolic plane: Circle through two points not unique In space there is only one line through two points; Why isn’t this true of their projection? Contradiction?

71. 71 Images of lines in space All lines intersect the fronto-parallel horizon (projection of the equator) antipodally

72. 72

73. 73 image center twice focal length focal length

74. 74 d r 2f

75. 75 When taking into account image center and circle center, the constraint becomes: Line image constraint (x,y)(x,y) (u,v)(u,v)

76. 76 When taking into account image center and circle center, the constraint becomes: Line image constraint (x,y)(x,y) (u,v)(u,v) …and can be written as where  represents an imaginary circle

77. 77 Line image constraint

78. 78 Line image constraint Implies calibration by fitting plane to circle representations

79. 79 Interpretation: Absolute and calibrating conics absolute conic calibrating conic

80. 80 We have a system of representation for: –Image features –Real radii circles –Imaginary radii circles Conditions for coincidence Formula for angle of intersection Condition that a circle be a line image, absolute conic Summary up to now

81. 81 We have a system of representation for: –Image features –Real radii circles –Imaginary radii circles Conditions for coincidence Formula for angle of intersection Condition that a circle be a line image, absolute conic Summary up to now Questions?

82. 82 Uncalibrated cameras Applying the inverse stereographic proj- ection gives…...the correct ray through space When we try to take the inverse if the camera is uncalibrated… …the inverse does not give the correct ray in space What is the transformation between the uncalibrated and calibrated points? The transformation is a linear one, i.e. y = A x … and it is a member of the Lorentz group O(3,1) k ° s = s ° k´ k´ linear

83. 83 Why should it be linear? Choose an arbitrary circle and find its inverse stereographic image

84. 84 Why should it be linear? Translate the circle; find the inverse stereographic image of the translated circle

85. 85 Why should it be linear? The translation in the plane induces a trans- formation of the sphere which preserves planes

86. 86 Uncalibrated cameras This argument applies to scaling, rotation and translation Thus a similarity transformation in the plane induces some projective linear transformation A of circle space It also sends any point satisfying to some point satisfying

87. 87 Sphere preserving transformations where The set of all such matrices is closed under matrix multi- plication, inversion and contains the identity; it is a group

88. 88 Lorentz and orthogonal groups Orthogonal group (in 4-dimensions) Lorentz group where The set of all such matrices is closed under matrix multi- plication, inversion and contains the identity; it is a group

89. 89 The Lorentz group 4 connected components (this is only a sketch)

90. 90 The Lorentz group orientation reversing orientatation preserving

91. 91 The Lorentz Lie group A(0) = I Suppose we have a curve satisfying:

92. 92 The Lorentz Lie group Implying: Differentiate both sides of to obtain:

93. 93 The Lorentz Lie group

94. 94 The Lorentz Lie group ex p For any matrix Lie group, a local one-to-one map from its Lie algebra back to the Lie group is given by the exponential map.

95. 95 Infinitessimal generators of the Lorentz group Rotations about the x-axis y-axisz-axis Generated by skew-symmetric matrices:

96. 96 Infinitessimal generators of the Lorentz group Translations along the x-axis y-axis Scaling about the origin Generated by :

97. 97 Lorentz group consistently transforms circle space One last property of Lorentz transformations is that they transform representations of circles consistent with the transformations of image points

98. 98 Lorentz group consistently transforms circle space One last property of Lorentz transformations is that they transform representations of circles consistent with the transformations of image points Questions?

99. 99 Inverting the projection With insight into properties of parabolic projections, let’s reconsider the problem of inverting an uncalibrated projection Recall that we can decompose the parabolic projection as: s n s is stereographic projection & n is projection to the sphere

100. 100 Inverting the projection With insight into properties of parabolic projections, let’s reconsider the problem of inverting an uncalibrated projection Recall that we can decompose the parabolic projection as: s n k is a calibration transformation k

101. 101 Inverting the projection However we now know that there exists some projective linear k´ such that s  k´ = k  s

102. 102 Inverting the projection We have the points in the plane and their inverse stereographic images k(x)k(x) x s -1 (k(x)) s -1 (x)

103. 103 Inverting the projection We have the points in the plane and their inverse stereographic images k(x)k(x) x s -1 (k(x)) s -1 (x) Problem: obtain s -1 (x) as a linear transformation of s -1 (k(x)).

104. 104 Inverting the projection We have the points in the plane and their inverse stereographic images k(x)k(x) x s -1 (k(x)) s -1 (x) Problem: obtain s -1 (x) as a linear transformation of s -1 (k(x)). Knowns: k(x) Unknowns: x, k Non-linear in unknown: = s -1 (k -1 (k(x))) s -1 (x) = s -1 (k -1 (k(x))) Knowns: k(x) Unknowns: x, k Non-linear in unknown: = s -1 (k -1 (k(x))) s -1 (x) = s -1 (k -1 (k(x)))

105. 105 Inverting the projection k and K´ commute about s -1 K´s -1 (x) = s -1 (k(x)) s -1 (x) k(x)k(x) x

106. 106 Inverting the projection Therefore the calibrated point is a linear transformation of the lifting of the uncalibrated point s -1 (x) = K´ -1 s -1 (k(x)) k(x)k(x) x K´s -1 (x) = s -1 (k(x))

107. 107 Inverting the projection Therefore the calibrated point is a linear transformation of the lifting of the uncalibrated point s -1 (x) = K´ -1 s -1 (k(x)) k(x)k(x) x K´s -1 (x) = s -1 (k(x)) Linear in unknown: s -1 (x) = Linear in unknown: s -1 (x) = K´ -1 s -1 (k(x))

108. 108 Linearization of the inverse projection Then the ray (in P 2 ), as a function of the uncalibrated image point is, is a linear transformation of the lifting PK´ -1 s -1 (k(x)) k(x)k(x) x P

109. 109 Linearization of the inverse projection Then the ray (in P 2 ), as a function of the uncalibrated image point is, is a linear transformation of the lifting PK´ -1 s -1 (k(x)) is equivalent to the perspective projection of the space point k(x)k(x) x PK´ -1 s -1 (k(x))

110. 110 Linearization of the inverse projection Then the ray (in P 2 ), as a function of the uncalibrated image point is, is a linear transformation of the lifting K´ -1 is and unknown but linear transformation (and can be absorbed into linear constraints) s -1 (x) is a non-linear but known transformation k(x)k(x) x PK´ -1 s -1 (k(x))

111. 111 uncalibrated rays calibrated rays We do not claim that there is a linear trans- formation from uncalibrated RAYS (i.e. elements of P 2 ) to calibrated RAYS (elements of P 2 )

112. 112 calibrated rays uncalibrated points Instead, we claim that there is a linear trans- formation from uncalibrated lifted image points (i.e. elements of P 3 ) to calibrated RAYS (elements of P 2 )

113. 113 Calibration transformation

114. 114 Calibration transformation on the absolute conic

115. 115 Calibration transformation on the absolute conic The point  is sent to the origin (0,0,0,1) in P 3 in The origin is in the null- space of the projection P Hence PK´ -1  = 0

116. 116 Circle space 1.Representation of point features 2.Conditions for incidence, etc. 3.Line image constraint

117. End of Part II Questions?

118. 118 Outline of Part III 1.The parabolic catadioptric fundamental matrix 2.Self-calibration 3.Kruppa equations trivially satisfied 4.Planar homographies & self-calibration 5.Multiple view geometry 6.Infinitessimal motions 7.Conformal rectification

119. 119 Deriving the parabolic epipolar constraint Suppose two views are separated by a rotation R and translation t. Given a point X in space, what constraint must the image points p 1 and p 2 satisfy?

120. 120 Deriving the parabolic epipolar constraint If we know the calibrated rays, then they are known to satisfy the epipolar constraint for perspective cameras (C. Longuet-Higgins)

121. 121 If the image points are uncalibrated, then we know that the calibrated rays are linearly related to the uncalibrated liftings Deriving the parabolic epipolar constraint

122. 122 F (4  4 parabolic fundamental matrix) Deriving the parabolic epipolar constraint

123. 123 Deriving the parabolic epipolar constraint Consequently lifted image points satisfy a bilinear epipolar constraint

124. 124 Self-calibration To each view there is associated an IAC which are represented by  1 and  2 They are in the nullspaces of PK i -1 and so in the nullspaces of F and F T

125. 125 Self-calibration If  =  1 =  2 i.e., the intrinsic parameters are the same, then  can be uniquely recovered from the intersection of the nullspaces: = Unless in which case:

126. 126 A characterization of parabolic fundamental matrices Recall that a 3  3 matrix E is an essential matrix if and only if for some U, V in SO(3) Claim: A 4  4 matrix F is a parabolic fundamental matrix if and only if for some U, V in SO(3,1)

127. 127 ()() Simple proof

128. 128 ()()()() Simple proof

129. 129 Estimation Because we have a bilinear constraint (and in general multilinear constraints) many methods that apply to the estimation of structure and motion from multiple perspective images apply, with some exceptions, to parabolic cameras. –Normalized epipolar constraint can be minimized –Unfortunately no equivalent to the 8/7-point algorithm (averaging Lorentzian singular values does not minimize Frobenius norm) –RANSAC and other robust methods apply –Structure estimation identical to perspective case once calibrated –Robust to modest deviations from ideal assumptions (e.g., non- aligned mirror, non-parabolic mirrors, etc.)

130. 130 Two view example Given these two views with corresponding points estimate the parabolic fundamental matrix

131. 131

132. 132

133. 133 epipolar circle two epipoles

134. 134  1  2  (  1,  2 )

135. 135

136. 136 δ δ

137. 137  1  2 A consequence of this is that the epipolar geometry is completely determined by the two epipoles in each image and the angle  Therefore the epipolar geometry has 9 parameters whereas the motion (5) and intrinsics for each view (6) total 11. 2-parameter ambiguity.

138. 138

139. 139

140. 140

141. 141

142. 142

143. 143 What is the ambiguity? We showed that the the epipolar geometry is determined by nine parameters, and the motion and camera parameters by eleven, demonstrating that there is a two-parameter ambiguity. Meaning for any two images there is a two-parameter family of possible reconstructions giving rise to the images. What is this family? Is it closed under some subset of projective transformations?

144. 144 This is your house

145. 145 This is your house on a parabolic mirror

146. 146 This is your house on drugs i.e. this is the ambiguity in the reconstruction of a house; the ambiguity is not projective

147. End of Part III Questions?

148. 148 Outline of Part IV 1.Group-theoretic analysis of the parabolic fundamental matrix 2.Quotient spaces of bilinear forms (parabolic fundamental matrices and essential matrices) 3.Essential harmonic transform

149. 149 The space of parabolic fundamental matrices What is its structure? Is it a manifold? How many degrees-of- freedom does it have? What ambiguities are there in motion estimation?

150. 150 Group theoretic analysis of bilinear constraints Let’s examine the LSVD characterization of parabolic fundamental matrices: implies fundamental matrices are closed under left or right multiplication by Lorentz transformations, i.e. is also a parabolic fundamental matrix. Note: the same reasoning applies to essential matrices.

151. 151 Thus SO(3,1)  SO(3,1) acts upon the set of fundamental matrices F SO(3,1)  SO(3,1) (U,V) The action of SO(3,1)  SO(3,1) on

152. 152 F e = (I, I) The action of SO(3,1)  SO(3,1) on The identity of the group induces the identity map

153. 153 The action is associative F The action of SO(3,1)  SO(3,1) on g · h = (U 1 U 2,V 1 V 2 ) g = (U 1,V 1 ) h = (U 2,V 2 )

154. 154 The action is (left) associative F g · h = (U 1 U 2,V 1 V 2 ) g = (U 1,V 1 ) h = (U 2,V 2 ) The action of SO(3,1)  SO(3,1) on

155. 155 F1F1 g The action of SO(3,1)  SO(3,1) on F2F2 The action is transitive: for every F 1 & F 2 there exists some g taking F 1 to F 2

156. 156 F SO(3,1)  SO(3,1) parameterizes With the action , SO(3,1)  SO(3,1) parameterizes 

157. 157 F Because of transitivity, the parameterization is surjective (onto); there is a g mapping F to F´ F´F´ SO(3,1)  SO(3,1) parameterizes

158. 158 Since SO(3) is itself parameterized by  is parameterized by ex p parameterizes (X,Y)

159. 159 In fact since  is surjective in SO(3,1), so then is the parameterization of  parameterizes

160. 160 F The paramaterization may be redundant; e.g., more than one group element may map F to F g3g3 g2g2 g1g1 Parameterization not one-to-one

161. 161 F So… what elements leave F invariant? Call it H F g HFHF The set H F

162. 162 F At the very least it contains the identity The set H F

163. 163 F Also H F is closed under (i) composition g  h h g The set H F

164. 164 F Also H F is closed under and (ii) inversion g -1 g The set H F

165. 165 F Hence H F is a subgroup It is called the isotropy subgroup The isotropy subgroup HFHF

166. 166 F Multiply every element of H F by an element g Cosets of the isotropy subgroup g h2h2 h1h1 gh2gh2 gh1gh1

167. 167 F What we obtain is a translation of H F by g; a coset of H F Cosets of the isotropy subgroup HFHF g  H F

168. 168 F F1F1 F2F2 Claim: any two elements of the coset g  H F map F to the same fundamental matrix Cosets of the isotropy subgroup Claim: F 1 = F 2

169. 169 F F1F1 Since h 1 is in H F and by the associativity of the action, g and g · h 1 both send F to the same point Cosets of the isotropy subgroup

170. 170 F F1F1 The same reasoning applies to h 2 and so F 1 = F 2 Cosets of the isotropy subgroup F2F2

171. 171 F F1F1 The same reasoning applies to h 2 and so F 1 = F 2 Cosets of the isotropy subgroup F2F2

172. 172 F Consequently every coset is in one-to-one correspondence with a fundamental matrix Cosets of the isotropy subgroup HFHF g  H F h·Fh·F g·Fg·F h  H F

173. 173 F The cosets are pairwise disjoint and their union is all of SO(3,1)  SO(3,1); they form a partition Cosets of H F partition SO(3,1)  SO(3,1)

174. 174 F Quotient spaces The partition of a group into its cosets is called the quotient space

175. 175 Because of its one-to-one correspondence, the set of fundamental matrices inherits the structure of a quotient space The set of fundamental matrices form a quotient space

176. 176 The dimension of the quotient space is the difference in the dimensions of the Lie groups Quotient of Lie algebras are automatically manifolds

177. 177 The dimension of the quotient space is the difference in the dimensions of the Lie groups Quotient of Lie algebras are automatically manifolds

178. 178 The dimension of the quotient space is the difference in the dimensions of the Lie groups Quotient of Lie algebras are automatically manifolds 9123 = –

179. 179 All of these results also apply to essential matrices Instead, SO(3)  SO(3) acts on the set of essential matrices SO(3)  SO(3) HEHE

180. 180 Harmonic analysis of bilinear forms Is it just a novelty that essential matrices and parabolic fundamental matrices are quotient spaces? In other words, who cares? We believe the description as a quotient space is important for the following reasons: –Simple unifying geometric description of bilinear –Global (surjective) nowhere-singular parameterization –These spaces are now endowed with Fourier transforms

181. 181 The rotational harmonic transform Recall that the Fourier transform is a projection of functions on L 2 ( 0,  ): Similarly the rotational harmonic transform (RHT) is a projection of square integrable functions on SO(3) — denoted L 2 ( SO(3) ) — onto an orthonormal basis:

182. 182 The rotational harmonic transform Recall that the Fourier transform is a projection of functions on L 2 ( 0,  ): Similarly the rotational harmonic transform (RHT) is a projection of square integrable functions on SO(3) — denoted L 2 ( SO(3) ) — onto an orthonormal basis: “Wigner d-coefficients” Rotation invariant measure on SO(3)

183. 183 The rotational harmonic transform The rotational harmonic transform obeys a number of properties some of which are: –Limit of partial sums converge to function –Parseval equality –Shift theorem –Convolution theorem

184. 184 Functions on the quotient space To define a function on the space of essential matrices we take some function on SO(3)  SO(3) and require that it be constant on cosets of H E. Alternatively f equals its average over all cosets. Recall that the subgroup and the cosets are g H E

185. 185 The essential harmonic transform The essential harmonic transform is a projection of such a function onto the bi-rotational harmonics of SO(3)  SO(3): and because it is constant over the cosets it satisfies

186. 186 Applications Q: What can we do with an essential harmonic transform? A: Fast convolutions. Might it be possible to estimate an essential matrix via a convolution of two signals to obtain a kind of correlation value for all possible essential matrices? Is it possible to unite signal processing and geometry? To be continued…

187. 187 Given two parabolic catad-ioptric cameras, rectify the stereo pair, i.e., transform both images so that corresp-onding points lie on the same scanline. x1x1 x2x2

188. 188

189. 189

190. 190 2δ

191. 191 2δ

192. 192 Both Möbius transformations and the equivalent to homographies must preserve line images (circles) and are therefore insufficient What transformations can perform the rectification?

193. 193 Bipolar coordinate system

194. 194 θ

195. 195 r1r1 r2r2 r 1 / r 2 constant

196. 196 θ This is analytic (i.e., differentiable) and therefore conformal 1

197. 197 1 1

198. 198

199. 199

200. End of Part IV Questions?

201. 201 Two-view geometry of catadioptric cameras –Geyer & Daniilidis, “Mirrors in Motion” ICCV 2003 Single-view geometry of catadioptric cameras –Geyer & Daniilidis, “Catadioptric Projective Geometry” IJCV Dec. 2001 Epipolar geometry of central catadioptric cameras –Pajdla & Svoboda, IJCV 2002 Theory of Catadioptric Image Formation –Baker & Nayar, IJCV Complex analysis & inversive geometry Geometry of Complex Numbers by Hans Schwerdtfeger, Dover Visual Complex Analysis by Tristan Needham, Oxford University Press Inversion Theory and Conformal Mapping by David Blair, AMS

202. 202

203. 203

204. 204 Deriving the fixed viewpoint constraint ( x, f (x) )

205. 205 Deriving the fixed viewpoint constraint ( x, f (x) )

206. 206 Deriving the fixed viewpoint constraint

207. 207 Deriving the fixed viewpoint constraint

208. 208 Deriving the fixed viewpoint constraint – π / 2

209. 209 Unifying model of catadioptric projection & consequences: 1.Line images are conics 2.Conformality of stereographic projection 3.Indistinguishability of elliptic and hyperbolic projections 4.Inadequacy of projective plane for catadioptric systems

210. 210 Unifying model of catadioptric projection & consequences: 1.Line images are conics 2.Conformality of stereographic projection 3.Indistinguishability of elliptic and hyperbolic projections 4.Inadequacy of projective plane for catadioptric systems Circle space 1.Representation of point features 2.Conditions for incidence, etc. 3.Line image constraint

211. 211 Unifying model of catadioptric projection & consequences: 1.Line images are conics 2.Conformality of stereographic projection 3.Indistinguishability of elliptic and hyperbolic projections 4.Inadequacy of projective plane for catadioptric systems Circle space 1.Representation of point features 2.Conditions for incidence, etc. 3.Line image constraint Lorentz transformations 1.Linearization of projection formula 2.Conformal 3.Calibration subgroup 4.Decomposition into calibration transformation and rotation

212. 212 Unifying model of catadioptric projection & consequences: 1.Line images are conics 2.Conformality of stereographic projection 3.Indistinguishability of elliptic and hyperbolic projections 4.Inadequacy of projective plane for catadioptric systems Circle space 1.Representation of point features 2.Conditions for incidence, etc. 3.Line image constraint Lorentz transformations 1.Linearization of projection formula 2.Conformal 3.Calibration subgroup 4.Decomposition into calibration transformation and rotation Parabolic fundamental matrix 1.Bilinear epipolar constraint 2.Self-calibration in two views 3.Equal Lorentzian singular values 4.Trivial satisfying Kruppa equations

213. 213 Unifying model of catadioptric projection & consequences: 1.Line images are conics 2.Conformality of stereographic projection 3.Indistinguishability of elliptic and hyperbolic projections 4.Inadequacy of projective plane for catadioptric systems Circle space 1.Representation of point features 2.Conditions for incidence, etc. 3.Line image constraint Lorentz transformations 1.Linearization of projection formula 2.Conformal 3.Calibration subgroup 4.Decomposition into calibration transformation and rotation Parabolic fundamental matrix 1.Bilinear epipolar constraint 2.Self-calibration in two views 3.Equal Lorentzian singular values 4.Trivial satisfying Kruppa equations Group theoretic characterization 1.Quotient space of fundamental and essential matrices 2.Essential harmonic transform

214. 214 Unifying model of catadioptric projection & consequences: 1.Line images are conics 2.Conformality of stereographic projection 3.Indistinguishability of elliptic and hyperbolic projections 4.Inadequacy of projective plane for catadioptric systems Circle space 1.Representation of point features 2.Conditions for incidence, etc. 3.Line image constraint Lorentz transformations 1.Linearization of projection formula 2.Conformal 3.Calibration subgroup 4.Decomposition into calibration transformation and rotation Parabolic fundamental matrix 1.Bilinear epipolar constraint 2.Self-calibration in two views 3.Equal Lorentzian singular values 4.Trivial satisfying Kruppa equations Group theoretic characterization 1.Quotient space of fundamental and essential matrices 2.Essential harmonic transform Any questions?

215. 215 Two-view geometry of catadioptric cameras –Geyer & Daniilidis, “Mirrors in Motion” ICCV 2003 Single-view geometry of catadioptric cameras –Geyer & Daniilidis, “Catadioptric Projective Geometry” IJCV Dec. 2001 Epipolar geometry of central catadioptric cameras –Pajdla & Svoboda, IJCV 2002 Theory of Catadioptric Image Formation –Baker & Nayar, IJCV Omnidirectional vision in general –Baker & Nayar, Panoramic Vision Relating to complex geometry –Hans Schwerdtfeger Geometry of Complex Numbers, Dover –Tristan Needham, Visual Complex Analysis, Oxford University Press –David Blair, Inversion Theory and Conformal Mapping, AMS

216. 5 minute break