1. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 1 Structure and Motion from Uncalibrated Catadioptric Views Christopher Geyer and Kostas Daniilidis Christopher Geyer and Kostas Daniilidis of the GRASP Laboratory, University of Pennsylvania of the GRASP Laboratory, University of Pennsylvania

2. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 2 The big picture: Two views obtained from a parabolic catadioptric camera 1 are sufficient to perform a Euclidean reconstruction from point correspondences NO PROJECTIVE AMBIGUITY! Fine print: 1. with non-varying intrinsics; if intrinsics vary, three frames are necessary; it is assumed that aspect ratio is 1 and there is no skew.

3. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 3 Assumptions Parabolic mirror Aspect ratio known Skew known Camera and mirror aligned Orthographic projection

4. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 4 Outline Review –Related work –What’s a catadioptric sensor? –The projection model Theory –Define representation of image features: circle space –Define the catadioptric fundamental matrix Algorithm –Computation of the fundamental matrix –Reconstruction Experiment Conclusion & future work

5. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 5 Related work Nayar, Baker –theory of catadioptric image formation [ICCV `98] Geyer, Daniilidis –theory of projective geometry induced in catadioptric images [see December `01 IJCV, ECCV `00] Svoboda, Pajdla –description of epipolar geometry [ECCV `98] Kang –non-linear self-calibration [CVPR `00] general cata- dioptric 2,3 n-view

6. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 6 What’s a catadioptric camera? Catadioptric camera combines a mirror and lens. We investigate cameras with a parabolic mirror and orthographically projecting lens Mirror CCD Lens sample image

7. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 7 Projection model Setup: Parabola p with focus F. Let s be the image plane. Find the catadioptric projection Q of the point P. F P s p (Review)

8. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 8 Projection model (Review) F P s R Q The image of P is the orthographic projection of the intersection of the line through F and P with the parabola.

9. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 9 Projection model The formula for this projection is: (where (x,y,z) is the space point and (u,v) is the image point) Uh-oh! It’s not linear! How do we apply SFM algorithms? (Review)

10. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 10 Projection model But wait! Neither is the perspective projection formula: That’s why we use homo- geneous coordinates invented by Möbius and Feuerbach in 1827. Feuerbach Möbius (Review)

11. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 11 Projection model Using homogeneous coordinates we can write: Embedding image points in a higher dimensional space linearizes the projection equation (Review)

12. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 12 Projection model Can we perform a similar trick for the parabolic projection? Yes, represent points in a higher dimensional space, lying on a paraboloid. But first we review properties of line images. (Review)

13. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 13 What is the image of a line? (Review)

14. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 14 A circle! But what kind? Must have same dof as line images

15. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 15 One that intersects..

16. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 16 reduces 3 dof to 2 dof One that the calibrating intersects..conic antipodally Calibrating conic we call ω ΄ Calibrating conic we call ω ΄

17. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 17 Equation of calibrating conic: Equation of image of absolute conic Same radii; ratio of radii = i; both encode the intrinsics Projection model

18. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 18 Circle Space Is there a clever representation of the space of circles? Yes! And it uses the paraboloid! See Dan Pedoe’s Geometry (Theory)

19. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 19 Choose a point P NOT NECESSARILY THE SAME PARABOLOID

20. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 20 Find the cone with vertex P tangent to the paraboloid

21. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 21 Find the plane through the inters- ection of the cone and the paraboloid

22. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 22 Orthographically project the intersection: a circle centered at the shadow of P

23. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 23 What is the result of varying P?

24. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 24 Summary

25. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 25 Lifting of image points p p (lifting) ~

26. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 26 In other words… We add a fourth coordinate:

27. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 27 Summary In this “circle space” we can represent line images and image points How do we represent circles intersecting the calibrating conic antipodally? (Theory)

28. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 28 Collinear point representations correspond to coaxal circles

29. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 29 Coplanar point representations map to circles intersecting a circle antipodally

30. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 30 How does the plane change when the image plane translates?

31. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 31 Circle space What is the significance of this plane? As it turns out it is the polar plane of the point representation (Theory)

32. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 32 Absolute conic (Theory) Image of the absolute conic Calibrating conic Plane of antipodally intersecting circles

33. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 33 Projections of space points space point it’s image

34. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 34 Projections of space points image of the absolute conic space point it’s image

35. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 35 But uncalibrated

36. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 36 Transformations of Circle Space Linear transfor- mation

37. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 37 Step to linearization (Theory)

38. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 38 Linearization There is a 3 x 4 K such that (the perspective projection) in particular (Theory)

39. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 39 Linearization The image of the absolute conic is in the kernel of K (Theory)

40. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 40 Two views P p q

41. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 41 On to the epipolar constraint If p and q are projections of the same point in two catadioptric images associated with K 1 and K 2 then and are the perspective projections.

42. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 42 The epipolar constraint The perspective projections satisfy for some essential matrix E. Therefore this is the epipolar constraint for parabolic catadioptric cameras.

43. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 43 Fundamental matrix The 4 x 4 matrix has rank 2 and since it satisfies (Theory)

44. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 44 Computation of F The product is linear in the entries of F. Therefore F can be computed using SVD just like in the perspective case. (Algorithm)

45. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 45 Reconstruction algorithm Non-varying intrinsics: 1. Compute F from >= 16 point matches

46. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 46 Reconstruction algorithm Non-varying intrinsics: 1. Compute F from > 15 point matches 2. Ensure F is rank 2: project to rank 2 manifold (using SVD)

47. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 47 Reconstruction algorithm Non-varying intrinsics: 1. Compute F from > 15 point matches 2. Ensure F is rank 2: project to rank 2 manifold (using SVD) 3. Find ω by intersecting left and right nullspaces of F ~

48. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 48 Reconstruction algorithm Non-varying intrinsics: 1. Compute F from > 15 point matches 2. Ensure F is rank 2: project to rank 2 manifold (using SVD) 3. Find ω by intersecting left and right nullspaces of F 4. Compute E = K -T FK -1 (K found from ω) ~ ~

49. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 49 Reconstruction algorithm Non-varying intrinsics: 1. Compute F from > 15 point matches 2. Ensure F is rank 2: project to rank 2 manifold (using SVD) 3. Find ω by intersecting left and right nullspaces of F 4. Compute E = K -T FK -1 (K found from ω) 5. Project E to manifold of essential matrices (average its singular values) ~ ~

50. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 50 Reconstruction algorithm Non-varying intrinsics: 1. Compute F from > 15 point matches 2. Ensure F is rank 2: project to rank 2 manifold (using SVD) 3. Find ω by intersecting left and right nullspaces of F 4. Compute E = K -T FK -1 (K found from ω) 5. Project E to manifold of essential matrices (average its singular values) 6. Do Euclidean reconstruction with E ~ ~

51. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 51 Varying intrinsics Three views w/absolute conics ω 1, ω 2, ω 3 compute fundamental matrices between pairs of views: F 12, F 23, F 31 Intersect nullspaces of F 12 T and F 31 to find ω 1 Similarly for ω 2 and ω 3 ~~ ~ ~ ~ ~

52. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 52 Experiment First view

53. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 53 Experiment Second view same intrinsics

54. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 54 Experiment Generate point correspondences manually or automatically

55. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 55 Reconstruction

56. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 56 catadioptric fundamental matrix Conclusion We have defined a represent- ation of image features in a parabolic catadioptric image catadioptric fundamental matrixWe have shown that the catadioptric fundamental matrix yields a bilinear constraint on lifted image points Its left and right nullspaces encode the image of the absolute conic.

57. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 57 Conclusion (cont’d) Non-varying intrinsics: in two views create Euclidean reconstruction Varying intrinsics: three views are sufficient to compute Euclidean structure Though aspect ratio 1 and skew 0 is assumed, this still beats the perspective case with same assumption where there is a projective ambiguity

58. December 12 th, 2001C. Geyer/K. Daniilidis GRASP Laboratory Slide 58 Future Work Necessary and sufficient conditions on matrix F Critical motions Ambiguous surfaces Relaxing aspect ratio and skew assumption Hyperbolic mirrors Thanks!