1. Multiple View Geometry Marc Pollefeys University of North Carolina at Chapel Hill Modified by Philippos Mordohai

2. 2 Outline Computation of camera matrix P Introduction to epipolar geometry estimation Chapters 6 and 8 of “Multiple View Geometry in Computer Vision” by Hartley and Zisserman

3. 3 Pinhole camera

4. 4 Camera center Principal point Principal ray Camera anatomy

5. 5 Camera calibration

6. 6 Resectioning Resectioning: correspondence between 3D and image entities

7. 7 Basic equations Similar to last week’s H estimation

8. 8 minimal solution Over-determined solution  5½ correspondences needed (say 6) P has 11 dof, 2 independent eq./points n  6 points minimize subject to constraint Basic equations

9. 9 (i)Camera and points on a twisted cubic (ii)Points lie on plane or single line passing through projection center Degenerate configurations

10. 10 Scaling Data normalization Centroid at origin

11. 11 Extend DLT to lines (back-project line) (2 independent eq.) Line correspondences

12. 12 Geometric error Assume 3D points are more accurately known

13. 13 Gold Standard algorithm Objective Given n≥6 3D to 2D point correspondences {X i ↔x i ’}, determine the Maximum Likelihood Estimation of P Algorithm (i)Linear solution: (a)Normalization: (b)DLT (ii)Minimization of geometric error: using the linear estimate as a starting point minimize the geometric error: (iii)Denormalization: ~~ ~

14. 14 (i)Canny edge detection (ii)Straight line fitting to the detected edges (iii)Intersecting the lines to obtain the images corners typically precision <1/10 (HZ rule of thumb: 5 n constraints for n unknowns) Calibration example

15. 15 Errors in the image and in the world Errors in the world

16. 16 Minimize geometric error  impose constraint through parameterization  Image only  9   2n, otherwise  3n+9   5n Find best fit that satisfies skew s is zero pixels are square principal point is known complete camera matrix K is known Minimize algebraic error  assume map from param q  P=K[R|-RC], i.e. p=g(q)  minimize ||Ag(q)|| (9 instead of 12 parameters) Restricted camera estimation

17. 17 One only has to work with 12x12 matrix, not 2nx12 Optimization cost is independent of the number of correspondences Reduced measurement matrix

18. 18 Initialization Use general DLT Clamp values to desired values, e.g. s=0,  x =  y Note: can sometimes cause big jump in error Alternative initialization Use general DLT Impose soft constraints gradually increase weights Restricted camera estimation

19. 19 Calibrated camera, position and orientation unknown  Pose estimation 6 dof  3 points minimal (4 solutions in general) Exterior orientation (hand-eye coordination)

20. 20 Algebraic method minimizes 12 errors instead of 2n=396 Experimental evaluation

21. 21 ML residual error Example: n=197, =0.365, =0.37 Covariance estimation d: number of parameters

22. 22 Compute Jacobian of measured points in terms of camera parameters at ML solution, then (variance per parameter can be found on diagonal) (chi-square distribution =distribution of sum of squares) cumulative -1 Covariance for estimated camera Confidence ellipsoid for camera center:

23. 23

24. 24 short and long focal length Radial distortion

25. 25

26. 26

27. 27 Choice of the distortion function and center Computing the parameters of the distortion function (i)Minimize with additional unknowns (ii)Straighten lines Correction of distortion 2

28. 28 Placing virtual models in video unmodelled radial distortion Bundle adjustment needed to avoid drift of virtual object throughout sequence (Sony TRV900; miniDV) modelled radial distortion

29. 29 Tsai calibration Zhang calibration http://research.microsoft.com/~zhang/calib/ Z. Zhang. A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330-1334, 2000. Z. Zhang. Flexible Camera Calibration By Viewing a Plane From Unknown Orientations. International Conference on Computer Vision (ICCV'99), Corfu, Greece, pages 666-673, September 1999. Some typical calibration algorithms

30. 30 Related Topics Calibration from vanishing points Calibration from the absolute conic

31. 31 Outline Computation of camera matrix P Introduction to epipolar geometry estimation

32. 32 (i)Correspondence geometry: Given an image point x in the first image, how does this constrain the position of the corresponding point x’ in the second image? (ii)Camera geometry (motion): Given a set of corresponding image points {x i ↔x’ i }, i=1,…,n, what are the cameras P and P’ for the two views? (iii)Scene geometry (structure): Given corresponding image points x i ↔x’ i and cameras P, P’, what is the position of (their pre-image) X in space? Three questions:

33. 33 C,C’,x,x’ and X are coplanar The epipolar geometry

34. 34 What if only C,C’,x are known? The epipolar geometry

35. 35 All points on  project on l and l’ The epipolar geometry

36. 36 Family of planes  and lines l and l’ Intersection in e and e’ The epipolar geometry

37. 37 epipoles e,e’ = intersection of baseline with image plane = projection of projection center in other image = vanishing point of camera motion direction an epipolar plane = plane containing baseline (1-D family) an epipolar line = intersection of epipolar plane with image (always come in corresponding pairs) The epipolar geometry

38. 38 Example: converging cameras

39. 39 (simple for stereo  rectification) Example: motion parallel with image plane

40. 40 e e’ Example: forward motion