1. Geometry of Images Pinhole camera, projection A taste of projective geometry Two view geometry:  Homography  Epipolar geometry, the essential matrix  Camera calibration, the fundamental matrix Stereo vision: 3D shape reconstruction from two views Factorization: reconstruction from many views

2. Geometry of Images

3. Cameras Camera obscura dates from 15 th century First photograph on record shown in the book - 1822 Basic abstraction is the pinhole camera Current cameras contain a lens and a recording device (film, CCD) The human eye functions very much like a camera

4. Camera Obscura "Reinerus Gemma-Frisius, observed an eclipse of the sun at Louvain on January 24, 1544, and later he used this illustration of the event in his book De Radio Astronomica et Geometrica, 1545. It is thought to be the first published illustration of a camera obscura..." Hammond, John H., The Camera Obscura, A Chronicle

5. Why Not Use Pinhole Camera If pinhole is too big - many directions are averaged, blurring the image Pinhole too small diffraction effects blur the image Generally, pinhole cameras are dark, because a very small set of rays from a particular point hits the screen.

6. Lenses

7. Lenses collect light from a large hole and direct it to a single point Overcome the darkness of pinhole cameras But there is a price  Focus  Radial distortions  Chromatic abberations  … Pinhole is useful as a model

8. Pinhole Camera

9. Single View Geometry f ∏

10. Notation O – Focal center π – Image plane Z – Optical axis f – Focal length

11. Projection f x y Z X Y

12. Perspective Projection Homogeneous Coordinates

13. Orthographic Projection Projection rays are parallel Image plane is fronto-parallel ( orthogonal to rays) Focal center at infinity

14. Scaled Orthographic Projection Also called “weak perspective”

15. Pros and Cons of Projection Models Weak perspective has simpler math.  Accurate when object is small and distant.  Most useful for recognition of objects. Pinhole perspective much more accurate for scenes.  Used in structure from motion. When accuracy really matters, we must model the real camera  Use perspective projection with other calibration parameters (e.g., radial lens distortion)

16. World Cup 66: England-Germany

17. World Cup 66: Second View

18. World Cup 66: England-Germany Conclusion: no goal (missing 3 inches) (Reid and Zisserman, “Goal-directed video metrology”)

19. Euclidean Geometry Answers the question what objects have the same shape (= congruent) Same shapes are related by rotation and translation

20. Projective Geometry Answers the question what appearances (projections) represent the same shape Same shapes are related by a projective transformation

21. Perspective Distortion Where do parallel lines meet? Parallel lines meet at the horizon (“vanishing line”)

22. Line Perspective Pencil of rays Perspective mapping

23. Plane Perspective

25. Ideal points Projective transformation can map ∞ to a real point

26. Coordinates in Euclidean Space 0 1 2 3 ∞ Not in space

27. Coordinates in Projective Line -1 0 1 2 ∞ k(0,1) k(1,0) k(2,1) k(1,1) k(-1,1) Points on a line P 1 are represented as rays from origin in 2D, Origin is excluded from space “Ideal point”

28. Coordinates in Projective Plane k(0,0,1) k(x,y,0) k(1,1,1) k(1,0,1) k(0,1,1) “Ideal point”

29. 2D Projective Geometry: Basics A point: A line: we denote a line with a 3-vector Points and lines are dual: p is on l if Intersection of two lines: A line through two points:

30. Cross Product Every entry is a determinant of the two other entries Area of parallelogram bounded by u and v

31. Ideal points Q: How many ideal points are there in P 2 ? A: 1 degree of freedom family – the line at infinity

32. Projective Transformation (Homography) Any finite sequence of perspectivities is a projective transformation Projective transformations map lines to lines Represented by an invertible 3x3 linear transformation (up to scale), denote by H, or Given homography H, how does it operate on lines?

33. Rotation: Translation: Euclidean Transformations (Isometries)

34. Hierarchy of Transformations Isometry (Euclidean), Similarity, Affine, general linear Projective,

35. Invariants LengthAreaAnglesParallelism Isometry √√√√ Similarity × × (Scale) √√ Affine ×××√ Projective ××××

36. Perspective Projection Note: P and p are related by a scale factor, but it is a different factor for each point (depends on Z)

37. Two View Geometry When a camera changes position and orientation, the scene moves rigidly relative to the camera In two cases this results in homography:  Camera rotates around its focal point  The scene is planar In this case the mapping from one image to the second is one to one and depth cannot be recovered In the general case the induced motion is more complex and is captured by what is termed “epipolar geometry”

38. Camera Rotation (R is 3x3 non-singular)

39. Intuitively A sequence of two perspectivities Algebraically Planar Scenes Scene Camera 1 Camera 2

40. Two Views Related by Homography Two images are related by homography: One to one mapping from p to p’ H contains 8 degrees of freedom Given correspondences, each point determines 2 equations 4 points are required to recover H Depth cannot be recovered

41. General Case: Epipolar Lines epipolar line

42. Epipolar Plane epipolar plane epipolar line Baseline P O O’

43. Epipole Every plane through the baseline is an epipolar plane, and determines a pair of epipolar lines in the two images Two systems of epipolar lines are obtained, each system intersects in a point, the epipole The epipole is the projection of the center of the other camera epipolar plane epipolar lines Baseline O O’

45. Epipolar Lines epipolar plane epipolar line Baseline P O O’ To define an epipolar plane, we define the plane through the two camera centers O and O’ and some point P. This can be written algebraically (in some world coordinates as follows:

46. Epipolar Lines Set world coordinates around the first camera What to do with O’P? Every rotation changes the observed coordinate in the second image We need to de-rotate to make the second image plane parallel to the first Replacing by image points

47. Essential Matrix Denote this by: Then Define, then E is called the “essential matrix”

48. Essential Matrix E is rank 2. Its (right and left) null spaces are the two epipoles is linear and homogeneous in E, E can be recovered up to scale using 8 points The additional constraint detE=0 reduces the needed points to 7 In fact, there are only 5 degrees of freedom in E,  3 for rotation  2 for translation (up to scale), determined by epipole

49. Internal Calibration Camera parameters may be unknown: (c x,c y ) camera center, (a x,a y ) pixel dimensions, b skew Radial distortions are not accounted for

50. Fundamental Matrix F, the fundamental matrix, too is rank 2 F has 7 d.o.f. (9 entries, homogeneous, and detF=0)

51. Summary HomographyPerspectiveOrthographic Form ShapeOne-to-one (Group) Concentric epipolar lines Parallel epipolar lines D.o.f.8(5) 5 Eqs/pnt211 Minimal configuration 45+ (8, linear)4 DepthNoYes, up to scale No, third view required