December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 1 Structure and Motion from Uncalibrated Catadioptric Views Christopher Geyer and Kostas.

Slides:

Advertisements

Similar presentations

Epipolar Geometry.

Advertisements

More on single-view geometry

Lecture 11: Two-view geometry

3D reconstruction.

Two-View Geometry CS Sastry and Yang

Self-calibration.

Two-view geometry.

Recovering metric and affine properties from images

Camera calibration and epipolar geometry

A Generic Concept for Camera Calibration Peter Sturm and Srikumar Ramaligam Sung Huh CPSC 643 Individual Presentation 4 April 15, 2009.

Structure from motion.

Recovering metric and affine properties from images

Vision, Video and Virtual Reality Omnidirectional Vision Lecture 6 Omnidirectional Cameras CSC 59866CD Fall 2004 Zhigang Zhu, NAC 8/203A

1 Basic geometric concepts to understand Affine, Euclidean geometries (inhomogeneous coordinates) projective geometry (homogeneous coordinates) plane at.

3D Computer Vision and Video Computing Omnidirectional Vision Topic 11 of Part 3 Omnidirectional Cameras CSC I6716 Spring 2003 Zhigang Zhu, NAC 8/203A.

Epipolar geometry. (i)Correspondence geometry: Given an image point x in the first view, how does this constrain the position of the corresponding point.

Structure from motion. Multiple-view geometry questions Scene geometry (structure): Given 2D point matches in two or more images, where are the corresponding.

Uncalibrated Geometry & Stratification Sastry and Yang

Epipolar Geometry and the Fundamental Matrix F

Multiple View Geometry Marc Pollefeys University of North Carolina at Chapel Hill Modified by Philippos Mordohai.

Multiple-view Reconstruction from Points and Lines

3D reconstruction of cameras and structure x i = PX i x’ i = P’X i.

CAU Kiel DAGM 2001-Tutorial on Visual-Geometric 3-D Scene Reconstruction 1 The plan for today Leftovers and from last time Camera matrix Part A) Notation,

Multiple View Geometry

3D Computer Vision and Video Computing 3D Vision Lecture 14 Stereo Vision (I) CSC 59866CD Fall 2004 Zhigang Zhu, NAC 8/203A

The Pinhole Camera Model

Projected image of a cube. Classical Calibration.

Lec 21: Fundamental Matrix

Camera parameters Extrinisic parameters define location and orientation of camera reference frame with respect to world frame Intrinsic parameters define.

Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates.

Automatic Camera Calibration

Lecture 11 Stereo Reconstruction I Lecture 11 Stereo Reconstruction I Mata kuliah: T Computer Vision Tahun: 2010.

875: Recent Advances in Geometric Computer Vision & Recognition Jan-Michael Frahm Fall 2011.

Multi-view geometry.

Epipolar geometry The fundamental matrix and the tensor

1 Preview At least two views are required to access the depth of a scene point and in turn to reconstruct scene structure Multiple views can be obtained.

Projective cameras Motivation Elements of Projective Geometry Projective structure from motion Planches : –

Projective Geometry. Projection Vanishing lines m and n.

Lecture 04 22/11/2011 Shai Avidan הבהרה : החומר המחייב הוא החומר הנלמד בכיתה ולא זה המופיע / לא מופיע במצגת.

Multiview Geometry and Stereopsis. Inputs: two images of a scene (taken from 2 viewpoints). Output: Depth map. Inputs: multiple images of a scene. Output:

1 Formation et Analyse d’Images Session 7 Daniela Hall 25 November 2004.

Two-view geometry Epipolar geometry F-matrix comp. 3D reconstruction

Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics CS329 Amnon Shashua.

Affine Structure from Motion

Two-view geometry. Epipolar Plane – plane containing baseline (1D family) Epipoles = intersections of baseline with image planes = projections of the.

EECS 274 Computer Vision Affine Structure from Motion.

Mixing Catadioptric and Perspective Cameras

Computer vision: models, learning and inference M Ahad Multiple Cameras

776 Computer Vision Jan-Michael Frahm & Enrique Dunn Spring 2013.

Auto-calibration we have just calibrated using a calibration object –another calibration object is the Tsai grid of Figure 7.1 on HZ182, which can be used.

MASKS © 2004 Invitation to 3D vision Uncalibrated Camera Chapter 6 Reconstruction from Two Uncalibrated Views Modified by L A Rønningen Oct 2008.

Geometry Reconstruction March 22, Fundamental Matrix An important problem: Determine the epipolar geometry. That is, the correspondence between.

Uncalibrated reconstruction Calibration with a rig Uncalibrated epipolar geometry Ambiguities in image formation Stratified reconstruction Autocalibration.

Structure from Motion Paul Heckbert, Nov , Image-Based Modeling and Rendering.

Stereo March 8, 2007 Suggested Reading: Horn Chapter 13.

Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches : –

EECS 274 Computer Vision Projective Structure from Motion.

Lec 26: Fundamental Matrix CS4670 / 5670: Computer Vision Kavita Bala.

Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates.

55:148 Digital Image Processing Chapter 11 3D Vision, Geometry

René Vidal and Xiaodong Fan Center for Imaging Science

Two-view geometry Computer Vision Spring 2018, Lecture 10

Epipolar geometry.

3D Photography: Epipolar geometry

Omnidirectional epipolar geometry

3D reconstruction class 11

Uncalibrated Geometry & Stratification

Reconstruction.

Two-view geometry.

Two-view geometry.

Presentation transcript:

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 1 Structure and Motion from Uncalibrated Catadioptric Views Christopher Geyer and Kostas Daniilidis Christopher Geyer and Kostas Daniilidis of the GRASP Laboratory, University of Pennsylvania of the GRASP Laboratory, University of Pennsylvania

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 2 The big picture: Two views obtained from a parabolic catadioptric camera 1 are sufficient to perform a Euclidean reconstruction from point correspondences NO PROJECTIVE AMBIGUITY! Fine print: 1. with non-varying intrinsics; if intrinsics vary, three frames are necessary; it is assumed that aspect ratio is 1 and there is no skew.

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 3 Assumptions Parabolic mirror Aspect ratio known Skew known Camera and mirror aligned Orthographic projection

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 4 Outline Review –Related work –What’s a catadioptric sensor? –The projection model Theory –Define representation of image features: circle space –Define the catadioptric fundamental matrix Algorithm –Computation of the fundamental matrix –Reconstruction Experiment Conclusion & future work

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 5 Related work Nayar, Baker –theory of catadioptric image formation [ICCV `98] Geyer, Daniilidis –theory of projective geometry induced in catadioptric images [see December `01 IJCV, ECCV `00] Svoboda, Pajdla –description of epipolar geometry [ECCV `98] Kang –non-linear self-calibration [CVPR `00] general catadioptric 2,3 n-view

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 6 What’s a catadioptric camera? Catadioptric camera combines a mirror and lens. We investigate cameras with a parabolic mirror and orthographically projecting lens Mirror CCD Lens sample image

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 7 Projection model Setup: Parabola p with focus F. Let s be the image plane. Find the catadioptric projection Q of the point P. F P s p (Review)

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 8 Projection model (Review) F P s R Q The image of P is the orthographic projection of the intersection of the line through F and P with the parabola.

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 9 Projection model The formula for this projection is: (where (x,y,z) is the space point and (u,v) is the image point) Uh-oh! It’s not linear! How do we apply SFM algorithms? (Review)

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 10 Projection model But wait! Neither is the perspective projection formula: That’s why we use homogeneous coordinates invented by Möbius and Feuerbach in Feuerbach Möbius (Review)

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 11 Projection model Using homogeneous coordinates we can write: Embedding image points in a higher dimensional space linearizes the projection equation (Review)

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 12 Projection model Can we perform a similar trick for the parabolic projection? Yes, represent points in a higher dimensional space, lying on a paraboloid. But first we review properties of line images. (Review)

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 13 What is the image of a line? (Review)

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 14 A circle! But what kind? Must have same dof as line images

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 15 One that intersects..

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 16 reduces 3 dof to 2 dof One that the calibrating intersects..conic antipodally Calibrating conic we call ω ΄ Calibrating conic we call ω ΄

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 17 Equation of calibrating conic: Equation of image of absolute conic Same radii; ratio of radii = i; both encode the intrinsics Projection model

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 18 Circle Space Is there a clever representation of the space of circles? Yes! And it uses the paraboloid! See Dan Pedoe’s Geometry (Theory)

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 19 Choose a point P NOT NECESSARILY THE SAME PARABOLOID

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 20 Find the cone with vertex P tangent to the paraboloid

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 21 Find the plane through the intersection of the cone and the paraboloid

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 22 Orthographically project the intersection: a circle centered at the shadow of P

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 23 What is the result of varying P?

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 24 Summary

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 25 Lifting of image points p p (lifting) ~

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 26 In other words… We add a fourth coordinate:

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 27 Summary In this “circle space” we can represent line images and image points How do we represent circles intersecting the calibrating conic antipodally? (Theory)

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 28 Collinear point representations correspond to coaxal circles

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 29 Coplanar point representations map to circles intersecting a circle antipodally

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 30 How does the plane change when the image plane translates?

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 31 Circle space What is the significance of this plane? As it turns out it is the polar plane of the point representation (Theory)

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 32 Absolute conic (Theory) Image of the absolute conic Calibrating conic Plane of antipodally intersecting circles

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 33 Projections of space points space point it’s image

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 34 Projections of space points image of the absolute conic space point it’s image

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 35 But uncalibrated

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 36 Transformations of Circle Space Linear transfor- mation

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 37 Step to linearization (Theory)

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 38 Linearization There is a 3 x 4 K such that (the perspective projection) in particular (Theory)

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 39 Linearization The image of the absolute conic is in the kernel of K (Theory)

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 40 Two views P p q

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 41 On to the epipolar constraint If p and q are projections of the same point in two catadioptric images associated with K 1 and K 2 then and are the perspective projections.

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 42 The epipolar constraint The perspective projections satisfy for some essential matrix E. Therefore this is the epipolar constraint for parabolic catadioptric cameras.

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 43 Fundamental matrix The 4 x 4 matrix has rank 2 and since it satisfies (Theory)

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 44 Computation of F The product is linear in the entries of F. Therefore F can be computed using SVD just like in the perspective case. (Algorithm)

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 45 Reconstruction algorithm Non-varying intrinsics: 1. Compute F from >= 16 point matches

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 46 Reconstruction algorithm Non-varying intrinsics: 1. Compute F from > 15 point matches 2. Ensure F is rank 2: project to rank 2 manifold (using SVD)

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 47 Reconstruction algorithm Non-varying intrinsics: 1. Compute F from > 15 point matches 2. Ensure F is rank 2: project to rank 2 manifold (using SVD) 3. Find ω by intersecting left and right nullspaces of F ~

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 48 Reconstruction algorithm Non-varying intrinsics: 1. Compute F from > 15 point matches 2. Ensure F is rank 2: project to rank 2 manifold (using SVD) 3. Find ω by intersecting left and right nullspaces of F 4. Compute E = K -T FK -1 (K found from ω) ~ ~

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 49 Reconstruction algorithm Non-varying intrinsics: 1. Compute F from > 15 point matches 2. Ensure F is rank 2: project to rank 2 manifold (using SVD) 3. Find ω by intersecting left and right nullspaces of F 4. Compute E = K -T FK -1 (K found from ω) 5. Project E to manifold of essential matrices (average its singular values) ~ ~

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 50 Reconstruction algorithm Non-varying intrinsics: 1. Compute F from > 15 point matches 2. Ensure F is rank 2: project to rank 2 manifold (using SVD) 3. Find ω by intersecting left and right nullspaces of F 4. Compute E = K -T FK -1 (K found from ω) 5. Project E to manifold of essential matrices (average its singular values) 6. Do Euclidean reconstruction with E ~ ~

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 51 Varying intrinsics Three views w/absolute conics ω 1, ω 2, ω 3 compute fundamental matrices between pairs of views: F 12, F 23, F 31 Intersect nullspaces of F 12 T and F 31 to find ω 1 Similarly for ω 2 and ω 3 ~~ ~ ~ ~ ~

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 52 Experiment First view

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 53 Experiment Second view same intrinsics

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 54 Experiment Generate point correspondences manually or automatically

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 55 Reconstruction

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 56 catadioptric fundamental matrix Conclusion We have defined a representation of image features in a parabolic catadioptric image catadioptric fundamental matrixWe have shown that the catadioptric fundamental matrix yields a bilinear constraint on lifted image points Its left and right nullspaces encode the image of the absolute conic.

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 57 Conclusion (cont’d) Non-varying intrinsics: in two views create Euclidean reconstruction Varying intrinsics: three views are sufficient to compute Euclidean structure Though aspect ratio 1 and skew 0 is assumed, this still beats the perspective case with same assumption where there is a projective ambiguity

December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 58 Future Work Necessary and sufficient conditions on matrix F Critical motions Ambiguous surfaces Relaxing aspect ratio and skew assumption Hyperbolic mirrors Thanks!