Mixing Catadioptric and Perspective Cameras Peter Sturm INRIA Rhône-Alpes France

Introduction

Introduction

Introduction • Existing results - epipolar geometry between omnidirectional cameras - motion estimation - (self-) calibration - ...

Our Goals • Study the geometry of hybrid stereo systems (omnidirectional and perspective cameras) - epipolar geometry - trifocal tensors - plane homographies • Applications - motion estimation - calibration - self-calibration, calibration transfer - 3D reconstruction - ...

Plan  Camera models • Epipolar geometry of hybrid systems • Derivation of matching tensors • Applications • Conclusions

Camera Models Perspective and affine cameras

Camera Models Central catadioptric cameras • mirror (surface of revolution of a conic) • camera • virtual optical center

Camera Models Central catadioptric cameras • mirror (surface of revolution of a conic) • camera • virtual optical center • calibration

Camera Models Types of central catadioptric cameras • hyperbola + perspective camera • parabola + affine camera • ...

Plan • Camera models  Epipolar geometry of hybrid systems • Derivation of matching tensors • Applications • Conclusions

Epipolar Geometry epipolar plane d epipolar line d’ epipolar line q’ q epipole

Epipolar Geometry epipolar conic

Epipolar Geometry epipolar line epipole epipolar conic epipoles

Epipolar Geometry Example

Epipolar Geometry “ conic ~ F q’ “ There exists with concretely: Interpretation: epipolar line d’ q’ d q epipole epipolar conic epipoles purely perspective case

Epipolar Geometry There exists with Interpretation: “ conic ~ F q’ “ concretely: Interpretation: “lifted coordinates” epipolar line epipole epipolar conic epipoles

Epipolar Geometry Example

Epipolar Geometry BUT... Until now, linear epipolar relation only found for: • any combination of perspective, affine or para-catadioptric cameras (parabolic mirrors) Not yet for: • other omnidirectional cameras than para-catadioptric ones (e.g. based on hyperbolic mirrors)

Epipolar Geometry Special case: • combination of perspective and para-catadioptric cameras • epipolar conics are circles • F is of dimension 4x3 • the « lifted coordinates » are:

Epipolar Geometry Epipoles: • is of rank 2 • The epipole of the perspective camera is the right null-vector of F • F has a one-dimensional left null-space  the two epipoles of the catadioptric camera are the left null-vectors that are valid lifted coordinates (quadratic constraint):

Plan • Camera models • Epipolar geometry of hybrid systems  Derivation of matching tensors • Applications • Conclusions

Matching Tensors • Multi-linear relations between coordinates of correponding image points • Purely perspective case: derivation based on linear equations representing projections (3D  2D) • Here: equations for back-projection (2D  3D directions) - perspective cameras with

Matching Tensors • Linear equations representing back-projections - perspective cameras - para-catadioptric cameras • is of dimension 3x4 • it depends - on the mirror’s intrinsic parameters - on the affine camera’s intrinsic parameters

Matching Tensors • Putting the equations together

Matching Tensors • Putting the equations together This matrix has a kernel  its rank is lower than 6  its determinant is zero  bilinear equation on the coefficients of and

Matching Tensors • Straightforward extension to more than 2 views ...

Plan • Camera models • Epipolar geometry of hybrid systems • Derivation of matching tensors  Applications - Self-calibration of omnidirectional cameras from fundamental matrices - Calibration transfer from an omnidirectional to a perspective camera - Self-calibration of omnidirectional cameras from a plane homography • Conclusions

Applications Self-calibration of omnidirectional cameras from fundamental matrices: • para-catadioptric camera has 3 intrinsic parameters • representation as a 4-vector of homogeneous coordinates • this vector is in the left null-space of the fundamental matrix of this camera, defined with respect to any other camera (perspective, affine, catadioptric)  self-calibration is possible from two or more fundamental matrices

Applications Calibration transfer from an omnidirectional to a perspective camera: • input: - calibration of a para-catadioptric camera - fundamental matrix with a perspective camera  closed-form solution for the focal length of the perspective camera

Applications Self-calibration of omnidirectional cameras from a plane homography: • input: - plane homography H with a perspective camera  recovery of intrinsic parameters of para-catadioptric camera (given by null-vector of H)

Plan • Camera models • Epipolar geometry of hybrid systems • Derivation of matching tensors • Applications  Conclusions

Conclusions Open questions Perspectives • Multi-linear matching relations between perspective, affine and para-catadioptric cameras • Applications in calibration, self-calibration, motion estimation, 3D reconstruction, ... Open questions • Fundamental matrix etc. for hyper-catadioptric cameras ? • Plane homographies « for the inverse direction » ? Perspectives • Hybrid trifocal tensors for line images • Multi-view 3D reconstruction for hybrid systems

Mixing Catadioptric and Perspective Cameras Peter Sturm INRIA Rhône-Alpes France