1. 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective

2. Two-views geometry Outline Background: Camera, Projection models Necessary tools: A taste of projective geometry Two view geometry:  Planar scene (homography ).  Non-planar scene (epipolar geometry). 3D reconstruction (stereo).

3. Perspective Projection Origin (0,0,0) is the Focal center X,Y ( x,y ) axis are along the image axis (height / width). Z is depth = distance along the Optical axis f – Focal length

4. Coordinates in Projective Plane P 2 k(0,0,1) k(x,y,0) k(1,1,1) k(1,0,1) k(0,1,1) “Ideal point” Take R 3 –{0,0,0} and look at scale equivalence class (rays/lines trough the origin).

5. 2D Projective Geometry: Basics A point: A line: we denote a line with a 3-vector Line coordinates are homogenous Points and lines are dual: p is on l if Intersection of two lines/points

6. Cross Product in matrix notation [ ] x Hartley & Zisserman p. 581

7. 2D Projective Transformation Projectivity: An invertible mapping h:P 2  P 2 S.T: Homography. A 3x3 (non singular) invertible matrix acting on homogenous 3-vectores. Collineation A transformations that map lines to lines Hartley & Zisserman p. 32 4 names 3 definitions

8. 2D Projective Transformation H is defined up to scale 9 parameters 8 degrees of freedom Determined by 4 corresponding points how does H operate on lines? Hartley & Zisserman p. 32

9. Next Homography Objectives: Understand when&how to use homography End of review Students should be back in buisness Time line: 15 min (60 sec/slide)

10. Two-views geometry Outline Background: Camera, Projection Necessary tools: A taste of projective geometry Two view geometry:  Homography  Epipolar geometry, the essential matrix  Camera calibration, the fundamental matrix 3D reconstruction from two views (Stereo algorithms)

11. Two View Geometry When a camera changes position and orientation, the scene moves rigidly relative to the camera 3-D Scene u u’u’ Rotation + translation

12. Two View Geometry (simple cases) In two cases this results in homography: 1. Camera rotates around its focal point 2. The scene is planar Then:  Point correspondence forms 1:1mapping  depth cannot be recovered

13. Camera Rotation (R is 3x3 non-singular)

14. Planar Scenes Intuitively A sequence of two perspectivities Algebraically Need to show: Scene Camera 1 Camera 2

15. Summary: Two Views Related by Homography Two images are related by homography: One to one mapping from p to p’ H contains 8 degrees of freedom Given correspondences, each point determines 2 equations 4 points are required to recover H Depth cannot be recovered

16. Next Epipolar geometry Objectives: Essential & fundamental matrices End of Homographies. Students should understand:  Why in both cases we end with a homography Time line: 45 min (180 sec/slide)

17. The General Case: Epipolar Lines epipolar line

18. Epipolar Plane epipolar plane epipolar line Baseline P O O’

19. Epipole Every plane through the baseline is an epipolar plane It determines a pair of epipolar lines (one in each image) Two systems of epipolar lines are obtained Each system intersects in a point, the epipole The epipole is the projection of the center of the other camera epipolar plane epipolar lines Baseline O O’

20. Example

21. Epipolar Lines epipolar plane epipolar line Baseline P O O’ To define an epipolar plane, we define the plane through the two camera centers O and O’ and some point P. This can be written algebraically (in some world coordinates as follows:

22. Essential Matrix (algebraic constraint between corresponding image points) Set world coordinates around the first camera What to do with O’P? Every rotation changes the observed coordinate in the second image We need to de-rotate to make the second image plane parallel to the first Replacing by image points Other derivations Hartley & Zisserman p. 241

23. Essential Matrix (cont.) Denote this by: Then Define E is called the “essential matrix”

24. Properties of the Essential Matrix E is homogeneous Its (right and left) null spaces are the two epipoles 9 parameters Is linear E, E can be recovered up to scale using 8 points. Has rank 2. The constraint detE=0  7 points suffices In fact, there are only 5 degrees of freedom in E,  3 for rotation  2 for translation (up to scale), determined by epipole

25. Background The lens optical axis does not coincide with the sensor We model this using a 3x3 matrix the Calibration matrix Camera Internal Parameters or Calibration matrix

26. Camera Calibration matrix The difference between ideal sensor ant the real one is modeled by a 3x3 matrix K: (c x,c y ) camera center, (a x,a y ) pixel dimensions, b skew We end with

27. Fundamental Matrix F, is the fundamental matrix.

28. Properties of the Fundamental Matrix F is homogeneous Its (right and left) null spaces are the two epipoles 9 parameters Is linear F, F can be recovered up to scale using 8 points. Has rank 2. The constraint detF=0  7 points suffices

29. Epipolar Plane l’ l Baseline P O O’ Other derivations Hartley & Zisserman p. 223 x X’ e e’ e’

30. HomographyEpipolar Form ShapeOne-to-one mapConcentric epipolar lines D.o.f.88/5 F/E Eqs/pnt21 Minimal configuration 45+ (8, linear) Depth NoYes, up to scale Scene Planar (or no translation) 3D scene Two-views geometry Summary:

31. Next Stereo Objectives: Basic Terminology & triangulation. End of. Epipolar geometry Time line: 70 min (120 sec/slide)

32. Stereo Vision Objective: 3D reconstruction Input: 2 (or more) images taken with calibrated cameras Output: 3D structure of scene Steps:  Rectification  Matching  Depth estimation

33. Rectification Image Reprojection  reproject image planes onto common plane parallel to baseline Notice, only focal point of camera really matters (Seitz)

34. Rectification Any stereo pair can be rectified by rotating and scaling the two image planes (=homography) We will assume images have been rectified so  Image planes of cameras are parallel.  Focal points are at same height.  Focal lengths same. Then, epipolar lines fall along the horizontal scan lines of the images

35. Cyclopean Coordinates Origin at midpoint between camera centers Axes parallel to those of the two (rectified) cameras

36. Disparity The difference is called “disparity” d is inversely related to Z: greater sensitivity to nearby points d is directly related to b: sensitivity to small baseline

37. Next tasest of correspondence Objectives: The problem of correspondence (solutions in next class). End of. Stereo Time line: 80 min (60 sec/slide)

38. Main Step: Correspondence Search What to match?  Objects? More identifiable, but difficult to compute  Pixels? Easier to handle, but maybe ambiguous  Edges?  Collections of pixels (regions)?

39. Random Dot Stereogram Using random dot pairs Julesz showed that recognition is not needed for stereo

40. Random Dot in Motion

41. Finding Matches

42. SSD error disparity 1D Search More efficient Fewer false matches

43. Ordering

45. Comparison of Stereo Algorithms D. Scharstein and R. Szeliski. "A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms," International Journal of Computer Vision, 47 (2002), pp. 7-42. Ground truthScene

46. Results with window correlation Window-based matching (best window size) Ground truth

47. Scharstein and Szeliski

48. Graph Cuts (next class). Ground truthGraph cuts

49. Next class Stereo algorithms End of this class Time line: 100 min