Mixing Catadioptric and Perspective Cameras Peter Sturm INRIA Rhône-Alpes France
Introduction
Introduction
Introduction • Existing results - epipolar geometry between omnidirectional cameras - motion estimation - (self-) calibration - ...
Our Goals • Study the geometry of hybrid stereo systems (omnidirectional and perspective cameras) - epipolar geometry - trifocal tensors - plane homographies • Applications - motion estimation - calibration - self-calibration, calibration transfer - 3D reconstruction - ...
Plan Camera models • Epipolar geometry of hybrid systems • Derivation of matching tensors • Applications • Conclusions
Camera Models Perspective and affine cameras
Camera Models Central catadioptric cameras • mirror (surface of revolution of a conic) • camera • virtual optical center
Camera Models Central catadioptric cameras • mirror (surface of revolution of a conic) • camera • virtual optical center • calibration
Camera Models Types of central catadioptric cameras • hyperbola + perspective camera • parabola + affine camera • ...
Plan • Camera models Epipolar geometry of hybrid systems • Derivation of matching tensors • Applications • Conclusions
Epipolar Geometry epipolar plane d epipolar line d’ epipolar line q’ q epipole
Epipolar Geometry epipolar conic
Epipolar Geometry epipolar line epipole epipolar conic epipoles
Epipolar Geometry Example
Epipolar Geometry “ conic ~ F q’ “ There exists with concretely: Interpretation: epipolar line d’ q’ d q epipole epipolar conic epipoles purely perspective case
Epipolar Geometry There exists with Interpretation: “ conic ~ F q’ “ concretely: Interpretation: “lifted coordinates” epipolar line epipole epipolar conic epipoles
Epipolar Geometry Example
Epipolar Geometry BUT... Until now, linear epipolar relation only found for: • any combination of perspective, affine or para-catadioptric cameras (parabolic mirrors) Not yet for: • other omnidirectional cameras than para-catadioptric ones (e.g. based on hyperbolic mirrors)
Epipolar Geometry Special case: • combination of perspective and para-catadioptric cameras • epipolar conics are circles • F is of dimension 4x3 • the « lifted coordinates » are:
Epipolar Geometry Epipoles: • is of rank 2 • The epipole of the perspective camera is the right null-vector of F • F has a one-dimensional left null-space the two epipoles of the catadioptric camera are the left null-vectors that are valid lifted coordinates (quadratic constraint):
Plan • Camera models • Epipolar geometry of hybrid systems Derivation of matching tensors • Applications • Conclusions
Matching Tensors • Multi-linear relations between coordinates of correponding image points • Purely perspective case: derivation based on linear equations representing projections (3D 2D) • Here: equations for back-projection (2D 3D directions) - perspective cameras with
Matching Tensors • Linear equations representing back-projections - perspective cameras - para-catadioptric cameras • is of dimension 3x4 • it depends - on the mirror’s intrinsic parameters - on the affine camera’s intrinsic parameters
Matching Tensors • Putting the equations together
Matching Tensors • Putting the equations together This matrix has a kernel its rank is lower than 6 its determinant is zero bilinear equation on the coefficients of and
Matching Tensors • Straightforward extension to more than 2 views ...
Plan • Camera models • Epipolar geometry of hybrid systems • Derivation of matching tensors Applications - Self-calibration of omnidirectional cameras from fundamental matrices - Calibration transfer from an omnidirectional to a perspective camera - Self-calibration of omnidirectional cameras from a plane homography • Conclusions
Applications Self-calibration of omnidirectional cameras from fundamental matrices: • para-catadioptric camera has 3 intrinsic parameters • representation as a 4-vector of homogeneous coordinates • this vector is in the left null-space of the fundamental matrix of this camera, defined with respect to any other camera (perspective, affine, catadioptric) self-calibration is possible from two or more fundamental matrices
Applications Calibration transfer from an omnidirectional to a perspective camera: • input: - calibration of a para-catadioptric camera - fundamental matrix with a perspective camera closed-form solution for the focal length of the perspective camera
Applications Self-calibration of omnidirectional cameras from a plane homography: • input: - plane homography H with a perspective camera recovery of intrinsic parameters of para-catadioptric camera (given by null-vector of H)
Plan • Camera models • Epipolar geometry of hybrid systems • Derivation of matching tensors • Applications Conclusions
Conclusions Open questions Perspectives • Multi-linear matching relations between perspective, affine and para-catadioptric cameras • Applications in calibration, self-calibration, motion estimation, 3D reconstruction, ... Open questions • Fundamental matrix etc. for hyper-catadioptric cameras ? • Plane homographies « for the inverse direction » ? Perspectives • Hybrid trifocal tensors for line images • Multi-view 3D reconstruction for hybrid systems
Mixing Catadioptric and Perspective Cameras Peter Sturm INRIA Rhône-Alpes France