1. Computer Vision Calibration Marc Pollefeys COMP 256 Read F&P Chapter 2 Some slides/illustrations from Ponce, Hartley & Zisserman

2. Computer Vision 2 Jan 16/18-Introduction Jan 23/25CamerasRadiometry Jan 30/Feb1Sources & ShadowsColor Feb 6/8Linear filters & edgesTexture Feb 13/15Multi-View GeometryStereo Feb 20/22Optical flowProject proposals Feb27/Mar1Affine SfMProjective SfM Mar 6/8Camera CalibrationSegmentation Mar 13/15Springbreak Mar 20/22FittingProb. Segmentation Mar 27/29Silhouettes and Photoconsistency Linear tracking Apr 3/5Project UpdateNon-linear Tracking Apr 10/12Object Recognition Apr 17/19Range data Apr 24/26Final project Tentative class schedule

3. Computer Vision 3 Previously Hierarchy of 3D transformations Projective 15dof Affine 12dof Similarity 7dof Euclidean 6dof Intersection and tangency Parallellism of planes, Volume ratios, centroids, The plane at infinity π ∞ Angles, ratios of length The absolute conic Ω ∞ Volume

4. Computer Vision 4 Camera calibration Compute relation between pixels and rays in space ?

5. Computer Vision 5 Pinhole camera

6. Computer Vision 6 Pinhole camera model linear projection in homogeneous coordinates! homogeneous coordinates non-homogeneous coordinates

7. Computer Vision 7 Pinhole camera model

8. Computer Vision 8 Principal point offset principal point

9. Computer Vision 9 Principal point offset calibration matrix

10. Computer Vision 10 Camera rotation and translation ~

11. Computer Vision 11 Object motion

12. Computer Vision 12 Camera motion

13. Computer Vision 13 CCD camera

14. Computer Vision 14 General projective camera non-singular 11 dof (5+3+3) intrinsic camera parameters extrinsic camera parameters

15. Computer Vision 15 Camera matrix decomposition Finding the camera center (use SVD to find null-space) Finding the camera orientation and internal parameters (use RQ decomposition ~QR) Q R =( ) -1 = -1 -1 Q R (if only QR, invert) (for all X and λ  C must be camera center)

16. Computer Vision 16 Affine cameras

17. Computer Vision 17 Radial distortion Due to spherical lenses (cheap) Model: R R http://foto.hut.fi/opetus/260/luennot/11/atkinson_6-11_radial_distortion_zoom_lenses.jpg straight lines are not straight anymore pincushion dist. barrel dist.

18. Computer Vision 18 Radial distortion example

19. Computer Vision 19 Camera model Relation between pixels and rays in space ?

20. Computer Vision 20 Projector model Relation between pixels and rays in space (dual of camera) (main geometric difference is vertical principal point offset to reduce keystone effect) ?

21. Computer Vision 21 Meydenbauer camera vertical lens shift to allow direct ortho-photographs

22. Computer Vision 22 Action of projective camera on points and lines forward projection of line back-projection of line projection of point

23. Computer Vision 23 Action of projective camera on conics and quadrics back-projection to cone projection of quadric

24. Computer Vision 24 Resectioning

25. Computer Vision 25 Direct Linear Transform (DLT) rank-2 matrix

26. Computer Vision 26 Direct Linear Transform (DLT) Minimal solution Over-determined solution  5½ correspondences needed (say 6) P has 11 dof, 2 independent eq./points n  6 points minimize subject to constraint use SVD

27. Computer Vision 27 Singular Value Decomposition Homogeneous least-squares

28. Computer Vision 28 Degenerate configurations (i)Points lie on plane or single line passing through projection center (ii)Camera and points on a twisted cubic

29. Computer Vision 29 Scale data to values of order 1 1.move center of mass to origin 2.scale to yield order 1 values Data normalization

30. Computer Vision 30 Line correspondences Extend DLT to lines (back-project line) (2 independent eq.)

31. Computer Vision 31 Geometric error

32. Computer Vision 32 Gold Standard algorithm Objective Given n≥6 2D to 2D point correspondences {X i ↔x i ’}, determine the Maximum Likelyhood Estimation of P Algorithm (i)Linear solution: (a)Normalization: (b)DLT (ii)Minimization of geometric error: using the linear estimate as a starting point minimize the geometric error: (iii)Denormalization: ~~ ~

33. Computer Vision 33 Calibration example (i)Canny edge detection (ii)Straight line fitting to the detected edges (iii)Intersecting the lines to obtain the images corners typically precision <1/10 (H&Z rule of thumb: 5 n constraints for n unknowns)

34. Computer Vision 34 Errors in the world Errors in the image and in the world Errors in the image (standard case)

35. Computer Vision 35 Restricted camera estimation Minimize geometric error  impose constraint through parametrization Find best fit that satisfies skew s is zero pixels are square principal point is known complete camera matrix K is known Minimize algebraic error  assume map from param q  P=K[R|-RC], i.e. p=g(q)  minimize ||Ag(q)||

36. Computer Vision 36 Restricted camera estimation Initialization Use general DLT Clamp values to desired values, e.g. s=0,  x =  y Note: can sometimes cause big jump in error Alternative initialization Use general DLT Impose soft constraints gradually increase weights

37. Computer Vision 37

38. Computer Vision 38 Image of absolute conic

39. Computer Vision 39 A simple calibration device (i)compute H for each square (corners  (0,0),(1,0),(0,1),(1,1)) (ii)compute the imaged circular points H(1,±i,0) T (iii)fit a conic to 6 circular points (iv)compute K from  through cholesky factorization (≈ Zhang’s calibration method)

40. Computer Vision 40 Some typical calibration algorithms Tsai calibration Zhangs calibration http://research.microsoft.com/~zhang/calib/ Z. Zhang. A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330-1334, 2000. Z. Zhang. Flexible Camera Calibration By Viewing a Plane From Unknown Orientations. International Conference on Computer Vision (ICCV'99), Corfu, Greece, pages 666-673, September 1999. http://www.vision.caltech.edu/bouguetj/calib_doc/

41. Computer Vision 41 Sequential SfM Initialize motion from two images Initialize structure For each additional view –Determine pose of camera –Refine and extend structure Refine structure and motion

42. Computer Vision 42 Initial projective camera motion Choose P and P´compatible with F Reconstruction up to projective ambiguity (reference plane;arbitrary) Initialize motion Initialize structure For each additional view Determine pose of camera Refine and extend structure Refine structure and motion Same for more views? different projective basis

43. Computer Vision 43 Initializing projective structure Reconstruct matches in projective frame by minimizing the reprojection error Non-iterative optimal solution Initialize motion Initialize structure For each additional view Determine pose of camera Refine and extend structure Refine structure and motion

44. Computer Vision 44 Projective pose estimation Infere 2D-3D matches from 2D-2D matches Compute pose from (RANSAC,6pts) F X x Inliers: Initialize motion Initialize structure For each additional view Determine pose of camera Refine and extend structure Refine structure and motion

45. Computer Vision 45 Refining structure Extending structure 2-view triangulation (Iterative linear) Initialize motion Initialize structure For each additional view Determine pose of camera Refine and extend structure Refine structure and motion Refining and extending structure

46. Computer Vision 46 Refining structure and motion use bundle adjustment Also model radial distortion to avoid bias!

47. Computer Vision 47 Metric structure and motion Note that a fundamental problem of the uncalibrated approach is that it fails if a purely planar scene is observed (in one or more views) (solution possible based on model selection) use self-calibration (see next class)

48. Computer Vision 48 Dealing with dominant planes

49. Computer Vision 49 PPPgric HHgric

50. Computer Vision 50 Farmhouse 3D models (note: reconstruction much larger than camera field-of-view)

51. Computer Vision 51 Application: video augmentation

52. Computer Vision 52 Next class: Segmentation Reading: Chapter 14