1. Mixing Catadioptric and Perspective Cameras
Peter Sturm
INRIA Rhône-Alpes
France

2. Introduction

3. Introduction

8. Introduction • Existing results
- epipolar geometry between omnidirectional
cameras
- motion estimation
- (self-) calibration
- ...

9. Our Goals • Study the geometry of hybrid stereo systems
(omnidirectional and perspective cameras)
- epipolar geometry
- trifocal tensors
- plane homographies
• Applications
- motion estimation
- calibration
- self-calibration, calibration transfer
- 3D reconstruction
- ...

10. Plan  Camera models • Epipolar geometry of hybrid systems
• Derivation of matching tensors
• Applications
• Conclusions

11. Camera Models
Perspective and affine cameras

12. Camera Models Central catadioptric cameras
• mirror (surface of revolution of a conic)
• camera
• virtual optical center

13. Camera Models Central catadioptric cameras
• mirror (surface of revolution of a conic)
• camera
• virtual optical center
• calibration

14. Camera Models Types of central catadioptric cameras
• hyperbola + perspective camera
• parabola + affine camera
• ...

15. Plan • Camera models  Epipolar geometry of hybrid systems
• Derivation of matching tensors
• Applications
• Conclusions

16. Epipolar Geometry epipolar plane d epipolar line d’ epipolar line q’ q
epipole

17. Epipolar Geometry
epipolar conic

18. Epipolar Geometry epipolar line epipole epipolar conic epipoles

19. Epipolar Geometry Example

20. Epipolar Geometry “ conic ~ F q’ “ There exists with concretely:
Interpretation:
epipolar
line d’ q’ d q
epipole
epipolar conic
epipoles
purely perspective case

21. Epipolar Geometry There exists with Interpretation: “ conic ~ F q’ “
concretely:
Interpretation:
“lifted coordinates”
epipolar
line
epipole
epipolar conic
epipoles

22. Epipolar Geometry Example

23. Epipolar Geometry
BUT...
Until now, linear epipolar relation only found for:
• any combination of perspective, affine or
para-catadioptric cameras (parabolic mirrors)
Not yet for:
• other omnidirectional cameras than para-catadioptric
ones (e.g. based on hyperbolic mirrors)

24. Epipolar Geometry Special case:
• combination of perspective and para-catadioptric
cameras
• epipolar conics are circles
• F is of dimension 4x3
• the « lifted coordinates » are:

25. Epipolar Geometry Epipoles: • is of rank 2
• The epipole of the perspective camera is the right
null-vector of F
• F has a one-dimensional left null-space
 the two epipoles of the catadioptric camera are the
left null-vectors that are valid lifted coordinates
(quadratic constraint):

26. Plan • Camera models • Epipolar geometry of hybrid systems
 Derivation of matching tensors
• Applications
• Conclusions

27. Matching Tensors
• Multi-linear relations between coordinates of correponding
image points
• Purely perspective case: derivation based on linear
equations representing projections (3D  2D)
• Here: equations for back-projection (2D  3D directions)
- perspective cameras
with

28. Matching Tensors • Linear equations representing back-projections
- perspective cameras
- para-catadioptric cameras
• is of dimension 3x4
• it depends
- on the mirror’s intrinsic parameters
- on the affine camera’s intrinsic parameters

29. Matching Tensors
• Putting the equations together

30. Matching Tensors • Putting the equations together
This matrix has a kernel
 its rank is lower than 6
 its determinant is zero
 bilinear equation on the coefficients of and

31. Matching Tensors
• Straightforward extension to more than 2 views ...

32. Plan • Camera models • Epipolar geometry of hybrid systems
• Derivation of matching tensors
 Applications
- Self-calibration of omnidirectional cameras from
fundamental matrices
- Calibration transfer from an omnidirectional to a
perspective camera
- Self-calibration of omnidirectional cameras from a
plane homography
• Conclusions

33. Applications Self-calibration of omnidirectional cameras from
fundamental matrices:
• para-catadioptric camera has 3 intrinsic parameters
• representation as a 4-vector of homogeneous coordinates
• this vector is in the left null-space of the fundamental
matrix of this camera, defined with respect to any other
camera (perspective, affine, catadioptric)
 self-calibration is possible from two or more
fundamental matrices

34. Applications Calibration transfer from an omnidirectional to a
perspective camera:
• input:
- calibration of a para-catadioptric camera
- fundamental matrix with a perspective camera
 closed-form solution for the focal length of the
perspective camera

35. Applications Self-calibration of omnidirectional cameras from
a plane homography:
• input:
- plane homography H with a perspective camera
 recovery of intrinsic parameters of para-catadioptric
camera (given by null-vector of H)

36. Plan • Camera models • Epipolar geometry of hybrid systems
• Derivation of matching tensors
• Applications
 Conclusions

37. Conclusions Open questions Perspectives
• Multi-linear matching relations between
perspective, affine and para-catadioptric cameras
• Applications in calibration, self-calibration,
motion estimation, 3D reconstruction, ...
Open questions
• Fundamental matrix etc. for hyper-catadioptric cameras ?
• Plane homographies « for the inverse direction » ?
Perspectives
• Hybrid trifocal tensors for line images
• Multi-view 3D reconstruction for hybrid systems

38. Mixing Catadioptric and Perspective Cameras
Peter Sturm
INRIA Rhône-Alpes
France