Multiple View Geometry Marc Pollefeys University of North Carolina at Chapel Hill Modified by Philippos Mordohai

2 Outline Computation of camera matrix P Introduction to epipolar geometry estimation Chapters 7 and 9 of “Multiple View Geometry in Computer Vision” by Hartley and Zisserman Note (2011): the slides refer to chapter numbers from the previous edition of the book. I try to update them but they may be off by one.

3 Pinhole camera

4 Camera center Principal point Principal ray Camera anatomy

5 Camera calibration

6 Resectioning Resectioning: correspondence between 3D and image entities

7 Basic equations Similar to last week’s H estimation

8 minimal solution Over-determined solution  5½ correspondences needed (say 6) P has 11 dof, 2 independent eq./points n  6 points minimize subject to constraint Basic equations

9 (i)Camera and points on a twisted cubic (ii)Points lie on plane or single line passing through projection center Degenerate configurations

10 Scaling Data normalization Centroid at origin

11 Geometric error Assume 3D points are more accurately known

12 Gold Standard algorithm Objective Given n≥6 3D to 2D point correspondences {X i ↔x i ’}, determine the Maximum Likelihood Estimation of P Algorithm (i)Linear solution: (a)Normalization: (b)DLT: Form 2nx12 matrix A and solve for Ap=0, subject to ||p||=1. p is the singular vector of A corresponding to the smallest singular value. (ii)Minimization of geometric error: using the linear estimate as a starting point minimize the geometric error: (iii)Denormalization: ~~ ~

13 (i)Canny edge detection (ii)Straight line fitting to the detected edges (iii)Intersecting the lines to obtain the images corners typically precision <1/10 (HZ rule of thumb: 5 n constraints for n unknowns) Calibration example

14 Errors in the image and in the world points Errors in the world

15 Minimize geometric error  impose constraint through parameterization  Image only  9   2n, otherwise  3n+9   5n Find best fit that satisfies skew s is zero pixels are square principal point is known complete camera matrix K is known Minimize algebraic error  assume map from param q  P=K[R|-RC], i.e. p=g(q)  minimize ||Ag(q)|| (9 instead of 12 parameters) Restricted camera estimation

16 Initialization Use general DLT Clamp values to desired values, e.g. s=0,  x =  y Note: can sometimes cause big jump in error Alternative initialization Use general DLT Impose soft constraints gradually increase weights Restricted camera estimation

17 Calibrated camera, position and orientation unknown  Pose estimation 6 dof  3 points minimal (4 solutions in general) Exterior orientation (hand-eye calibration)

18 Algebraic method minimizes 12 errors instead of 2n=396 Experimental evaluation

19 ML residual error Example: n=197, =0.365, =0.37 Covariance estimation d: number of parameters

20 Compute Jacobian of measured points in terms of camera parameters at ML solution, then (variance per parameter can be found on diagonal) (chi-square distribution =distribution of sum of squares) cumulative -1 Covariance for estimated camera Confidence ellipsoid for camera center:

21

22 short and long focal length Radial distortion

23

24

25 Choice of the distortion function and center Computing the parameters of the distortion function (i)Minimize with additional unknowns (ii)Straighten lines Correction of distortion 2

26 Placing virtual models in video unmodelled radial distortion Bundle adjustment needed to avoid drift of virtual object throughout sequence (Sony TRV900; miniDV) modelled radial distortion

27 Tsai calibration Zhang calibration Z. Zhang. A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11): , Z. Zhang. Flexible Camera Calibration By Viewing a Plane From Unknown Orientations. International Conference on Computer Vision (ICCV'99), Corfu, Greece, pages , September Some typical calibration algorithms

28 Related Topics Calibration from vanishing points Calibration from the absolute conic

29 Outline Computation of camera matrix P Introduction to epipolar geometry estimation

30 (i)Correspondence geometry: Given an image point x in the first image, how does this constrain the position of the corresponding point x’ in the second image? (ii)Camera geometry (motion): Given a set of corresponding image points {x i ↔x’ i }, i=1,…,n, what are the cameras P and P’ for the two views? (iii)Scene geometry (structure): Given corresponding image points x i ↔x’ i and cameras P, P’, what is the position of (their pre-image) X in space? Three questions:

31 C,C’,x,x’ and X are coplanar The epipolar geometry

32 What if only C,C’,x are known? The epipolar geometry

33 All points on  project on l and l’ The epipolar geometry

34 Family of planes  and lines l and l’ Intersection in e and e’ The epipolar geometry

35 epipoles e,e’ = intersection of baseline with image plane = projection of projection center in other image = vanishing point of camera motion direction an epipolar plane = plane containing baseline (1-D family) an epipolar line = intersection of epipolar plane with image (always come in corresponding pairs) The epipolar geometry

36 Example: converging cameras

37 (simple for stereo  rectification) Example: motion parallel with image plane

38 e e’ Example: forward motion

39 The fundamental matrix F algebraic representation of epipolar geometry we will see that mapping is a (singular) correlation (i.e. projective mapping from points to lines) represented by the fundamental matrix F

40 The fundamental matrix F correspondence condition The fundamental matrix satisfies the condition that for any pair of corresponding points x↔x ’ in the two images

41 The fundamental matrix F F is the unique 3x3 rank 2 matrix that satisfies x’ T Fx=0 for all x↔x’ (i)Transpose: if F is fundamental matrix for (P,P’), then F T is fundamental matrix for (P’,P) (ii)Epipolar lines: l’=Fx & l=F T x’ (iii)Epipoles: on all epipolar lines, thus e’ T Fx=0,  x  e’ T F=0, similarly Fe=0 (iv)F has 7 d.o.f., i.e. 3x3-1(homogeneous)-1(rank2) (v)F is a correlation, projective mapping from a point x to a line l’=Fx (not a proper correlation, i.e. not invertible)

42 Projective transformation and invariance Derivation based purely on projective concepts F invariant to transformations of projective 3-space unique not unique canonical form

43 The essential matrix ~fundamental matrix for calibrated cameras (remove K) 5 d.o.f. (3 for R; 2 for t up to scale) E is essential matrix if and only if two singular values are equal (and third=0)

44 Four possible reconstructions from E (only one solution where points is in front of both cameras)