1. Two-views geometry Outline Background: Camera, Projection Necessary tools: A taste of projective geometry Two view geometry:  Homography  Epipolar geometry, the essential matrix  Camera calibration, the fundamental matrix 3D reconstruction (Stereo algorithms) next week. Many of the slides are courtesy of Prof. Ronen Basri

2. 3-D Scene u u’u’ What can 2 images tell us about …. Faugeras et. al. ECCV 92 Objective

3. 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective

4. World Cup 66: England-Germany

5. World Cup 66: Second View

6. World Cup 66: England-Germany Conclusion: no goal (missing 3 inches) (Reid and Zisserman, “Goal-directed video metrology”)

7. Camera Obscura "Reinerus Gemma-Frisius, observed an eclipse of the sun at Louvain on January 24, 1544, and later he used this illustration of the event in his book De Radio Astronomica et Geometrica, 1545. It is thought to be the first published illustration of a camera obscura..." Hammond, John H., The Camera Obscura, A Chronicle

8. A few words about Cameras Camera obscura dates from 15 th century First photograph on record shown in the book - 1822 Basic abstraction is the pinhole camera Current cameras contain a lens and a recording device (film, CCD, CMOS) The human eye functions very much like a camera

9. Ideal Lenses Lens acts as a pinhole (for 3D points at the focal depth).

10. Regular Lenses E.g., the cameras in our lab. To learn more on lens-distortion see Hartley & Zisserman Sec. 7.4 p.189. Not part of this class.

11. Pinhole Camera

12. Single View Geometry f ∏

13. Notation O – Focal center π – Image plane Z – Optical axis f – Focal length

14. Projection f x y Z X Y

15. Perspective Projection Origin (0,0,0) is the Focal center X,Y ( x,y ) axis are along the image axis (height / width). Z is depth = distance along the Optical axis f – Focal length

16. Orthographic Projection Projection rays are parallel Image plane is fronto-parallel ( orthogonal to rays) Focal center at infinity

17. Scaled Orthographic Projection Also called “weak perspective”

18. Pros and Cons of Projection Models Weak perspective has simpler math.  Accurate when object is small and distant.  Useful for object recognition. Pinhole perspective much more accurate.  Used in structure from motion. When accuracy really matters (SFM), we must model the real camera (exact imaging processes):  Perspective projection, calibration parameters (later), and all other issues (radial distortion).

19. Two-views geometry Outline Background: Camera, Projection Necessary tools: A taste of projective geometry Two view geometry:  Homography  Epipolar geometry, the essential matrix  Camera calibration, the fundamental matrix 3D reconstruction from two views (Stereo algorithms) Hartley & Zisserman: Sec. 2 Proj. Geom. of 2D. Sec. 3 Proj. Geom. of 3D.

20. Reading Hartley & Zisserman: Sec. 2 Proj. Geo. of 2D: 2.1- 2.2.3 point lines in 2D 2.3 -2.4 transformations 2.7 line at infinity Sec. 3 Proj. Geo. of 3D. 3.1 – 3.2 point planes & lines. 3.4 transformations

21. Euclidean Geometry is good for questions like: what objects have the same shape (= congruent) Same shapes are related by rotation and translation Why projective Geometry (Motivation)

22. Why Projective Geometry (Motivation) Answers the question what appearances (projections) represent the same shape Same shapes are related by a projective transformation

23. Where do parallel lines meet? Parallel lines meet at the horizon (“vanishing line”) Why Projective Geometry (Motivation)

24. Coordinates in Euclidean Space 0 1 2 3 ∞ Not in space

25. Coordinates in Projective Line P 1 -1 0 1 2 ∞ k(0,1) k(1,0) k(2,1) k(1,1) k(-1,1) Points on a line P 1 are represented as rays from origin in 2D, Origin is excluded from space “Ideal point”

26. Coordinates in Projective Plane P 2 k(0,0,1) k(x,y,0) k(1,1,1) k(1,0,1) k(0,1,1) “Ideal point” Take R 3 –{0,0,0} and look at scale equivalence class (rays/lines trough the origin).

27. Projective Line vs. the Real Line -1 0 1 2 ∞ k(0,1) k(1,0) k(2,1) k(1,1) k(-1,1) “Ideal point” P1P1 RSymbol R^2 – {0,0}The real lineSpace Equivalence classes (2D “rays”) points Objects (points) Intersection with line y=1 Realization

28. Projective Plane vs Euclidian plane k(0,0,1) k(x,y,0) k(1,1,1) k(1,0,1) k(0,1,1) “Ideal line” P2P2 R2R2 Symbol R 3 – {0,0,0}The real planeSpace Equivalence classes (3D rays) point Objects (points) Intersection with plane z=1 Realization

29. 2D Projective Geometry: Basics A point: A line: we denote a line with a 3-vector Line coordinates are homogenous Points and lines are dual: p is on l if Intersection of two lines/points

30. Cross Product Every entry is a determinant of the two other entries Area of parallelogram bounded by u and v Hartley & Zisserman p. 581

31. Cross Product in matrix notation [ ] x Hartley & Zisserman p. 581

32. Example: Intersection of parallel lines Q: How many ideal points are there in P 2 ? A: 1 degree of freedom family – the line at infinity

33. Projective Transformations u u’u’

34. Transformations of the projective line Given a 2D linear transformation G:R 2  R 2 Study the induced transformation on the Equivalents classes. On the realization y=1 we get

35. Properties: 1. Invertible (T -1 exists) 2. Composable (T o G is a projective transformation) 3. Closed under composition Has 4 parameters 3 degrees of freedom Defined by 3 points Every point defines 1 constraint

36. Transformations of the projective line Pencil of rays Perspective mapping A perspective mapping is a projective transformation T:P 1  P 1 Perceptivity is a special projective mapping. Hartley & Zisserman p. 632 Lines connecting corresponding points are “concurrent”

37. Ideal points and projective transformations Projective transformation can map ∞ to a real point

38. Plane Perspective

40. 2D Projective Transformation Projectivity: An invertible mapping h:P 2  P 2 S.T: Homography. A 3x3 (non singular) invertible matrix acting on homogenous 3-vectores. Collineation A transformations that map lines to lines Hartley & Zisserman p. 32 4 names 3 definitions

41. 2D Projective Transformation H is defined up to scale 9 parameters 8 degrees of freedom Determined by 4 corresponding points how does H operate on lines? Hartley & Zisserman p. 32

42. Plane Perspective This mapping clearly maps lines to lines

43. Plane Perspective acting on conics Hartley & Zisserman p. 30 & 36 Not part of this class

44. Rotation: Translation: Hierarchy of Transformations Rigid (Isometry) Similarity Affine Projective Scale Hartley & Zisserman p. Sec. 2.4

45. Rotation: Translation: Euclidean Transformations (Isometries)

46. Hierarchy of Transformations Isometry (Euclidean), Similarity, Affine, general linear Projective,

47. Invariants LengthAreaAnglesParallelism Isometry √√√√ Similarity × × (Scale) √√ Affine ×××√ Projective ××××

48. Perspective Projection Note: P and p are related by a scale factor, but it is a different factor for each point (depends on Z)

49. Two-views geometry Outline Background: Camera, Projection Necessary tools: A taste of projective geometry Two view geometry:  Homography  Epipolar geometry, the essential matrix  Camera calibration, the fundamental matrix 3D reconstruction from two views (Stereo algorithms)

50. Two View Geometry When a camera changes position and orientation, the scene moves rigidly relative to the camera 3-D Scene u u’u’ Rotation + translation

51. 3-D Scene Rotation + translation u u’u’ Objective: find formulas that links corresponding points

52. Two View Geometry (simple cases) In two cases this results in homography: 1. Camera rotates around its focal point 2. The scene is planar Then:  Point correspondence forms 1:1mapping  depth cannot be recovered

53. Camera Rotation (R is 3x3 non-singular)

54. Planar Scenes Intuitively A sequence of two perspectivities Algebraically Need to show: Scene Camera 1 Camera 2

55. Summary: Two Views Related by Homography Two images are related by homography: One to one mapping from p to p’ H contains 8 degrees of freedom Given correspondences, each point determines 2 equations 4 points are required to recover H Depth cannot be recovered

56. The General Case: Epipolar Lines epipolar line

57. Epipolar Plane epipolar plane epipolar line Baseline P O O’

58. Epipole Every plane through the baseline is an epipolar plane It determines a pair of epipolar lines (one in each image) Two systems of epipolar lines are obtained Each system intersects in a point, the epipole The epipole is the projection of the center of the other camera epipolar plane epipolar lines Baseline O O’

59. Example

60. Epipolar Lines epipolar plane epipolar line Baseline P O O’ To define an epipolar plane, we define the plane through the two camera centers O and O’ and some point P. This can be written algebraically (in some world coordinates as follows:

61. Essential Matrix (algebraic constraint between corresponding image points) Set world coordinates around the first camera What to do with O’P? Every rotation changes the observed coordinate in the second image We need to de-rotate to make the second image plane parallel to the first Replacing by image points Other derivations Hartley & Zisserman p. 241

62. Essential Matrix (cont.) Denote this by: Then Define E is called the “essential matrix”

63. Properties of the Essential Matrix E is homogeneous Its (right and left) null spaces are the two epipoles 9 parameters Is linear E, E can be recovered up to scale using 8 points. Has rank 2. The constraint detE=0  7 points suffices In fact, there are only 5 degrees of freedom in E,  3 for rotation  2 for translation (up to scale), determined by epipole

64. Background The lens optical axis does not coincide with the sensor We model this using a 3x3 matrix the Calibration matrix Camera Internal Parameters or Calibration matrix

65. Camera Calibration matrix The difference between ideal sensor ant the real one is modeled by a 3x3 matrix K: (c x,c y ) camera center, (a x,a y ) pixel dimensions, b skew We end with

66. Fundamental Matrix F, is the fundamental matrix.

67. Properties of the Fundamental Matrix F is homogeneous Its (right and left) null spaces are the two epipoles 9 parameters Is linear F, F can be recovered up to scale using 8 points. Has rank 2. The constraint detF=0  7 points suffices

68. HomographyEpipolar Form ShapeOne-to-one mapConcentric epipolar lines D.o.f.88/5 F/E Eqs/pnt21 Minimal configuration 45+ (8, linear) Depth NoYes, up to scale Scene Planar (or no translation) 3D scene Two-views geometry Summary: