1. Camera Geometry and Calibration Thanks to Martial Hebert

2. Basic transformations (reminder)…. Det(R) = +1

3. Basic transformations (reminder)….

4. P z x y  C p u v Retinal plane f Normalized image plane 1

6. Scale in x direction between world coordinates and image coordinates Skew of camera axes.  = 90 o if the axes are perpendicular Principal point = Image coordinates of the projection of camera origin on the retina Rotation between world coordinate system and camera Translation between world coordinate system and camera Scale in y direction between world coordinates and image coordinates Standard Perspective Camera Model

7. Alternate Notations 4 3 3x3 3x1 By rows:By blocks:By components: 3x33x1 Intrinsic parameter matrix Extrinsic parameter matrix Q: Is a given 3x4 matrix M the projection matrix of some camera? A: Yes, if and only if det(A) is not zero Q: Is the decomposition unique? A: There are multiple equivalent solutions

8. Applying the Projection Matrix Homogeneous coordinates of point in image Homogeneous coordinates of point in world Homogeneous vector transformation: p proportional to MP Computation of individual coordinates:

9. Observations: is the equation of a plane of normal direction a 1 From the projection equation, it is also the plane defined by: u = 0 Similarly: (a 2,b 2 ) describes the plane defined by: v = 0 (a 3,b 3 ) describes the plane defined by:  That is the plane passing through the pinhole (z = 0) Geometric Interpretation Projection equation:

10. u v a3a3 C Geometric Interpretation: The rows of the projection matrix describe the three planes defined by the image coordinate system a1a1 a2a2

11. p P Other useful geometric properties Q: Given an image point p, what is the direction of the corresponding ray in space? A: Q: Can we compute the position of the camera center  ? A:

12. Affine Cameras Example: Weak-perspective projection model Projection defined by 8 parameters Parallel lines are projected to parallel lines The transformation can be written as a direct linear transformation 2x4 projection matrix 2x2 intrinsic parameter matrix 2x3 matrix = first 2 rows of the rotation matrix between world and camera frames First 2 components of the translation between world and camera frames Note: If the last row is the coordinates equations degenerate to: The mapping between world and image coordinates becomes linear. This is an affine camera.

16. Calibration: Recover M from scene points P 1,..,P N and the corresponding projections in the image plane p 1,..,p N In other words: Find M that minimizes the distance between the actual points in the image, p i, and their predicted projections MP i Problems: The projection is (in general) non-linear M is defined up to an arbitrary scale factor

17. Write relation between image point, projection matrix, and point in space: Write non-linear relations between coordinates: Make them linear: The math for the calibration procedure follows a recipe that is used in many (most?) problems involving camera geometry, so it’s worth remembering:

18. Put all the relations for all the points into a single matrix: Solve by minimizing: Subject to: Write them in matrix form:

19. Slight digression: Homogeneous Least Squares Suppose that we want to estimate the best vector of parameters X from matrices V i computed from input data such that VX i = 0 We can do this by minimizing: Since V=0 is a trivial solution, we need to constrain the magnitude of V to be non-zero, for example: |V| = 1. The problem becomes: The key result (which we will use 50 times in this class) is: For any symmetric matrix V, the minimum of X T VX is reached at X = eigenvector of V corresponding to the smallest eigenvalue.

20. P1P1 z x y PiPi (ui,vi)(ui,vi) (u1,v1)(u1,v1) MP i Calibration

23. We can follow exactly the same recipe with non-linear distortion: Write non-linear relations between coordinates: Make them linear:

24. Put all the relations for all the points into a single matrix: Solve by minimizing: Subject to: Write them in matrix form:

25. From Faugeras

28. p

29. http://www.vision.caltech.edu/bouguetj/calib_doc/htmls/ links.html http://www.ai.sri.com/~konolige/svs/svs.htm MATLAB calibration toolbox Intel OpenCV calibration package

30. Radiometric Calibration Important preprocessing step for many vision and graphics algorithms Use a color chart with precisely known reflectances. Irradiance = const * Reflectance Pixel Values 3.1%9.0%19.8%36.2%59.1%90% Use more camera exposures to fill up the curve. Method assumes constant lighting on all patches and works best when source is far away (example sunlight). Unique inverse exists because g is monotonic and smooth for all cameras. 0 255 01 g ? ?

31. Measured Response Curves of Cameras [Grossberg, Nayar]

32. Dark Current Noise Subtraction Dark current noise is high for long exposure shots To remove (some) of it: Calibrate the camera (make response linear) Capture the image of the scene as usual Cover the lens with the lens cap and take another picture Subtract the second image from the first image

33. Dark Current Noise Subtraction Original image + Dark Current Noise Image with lens cap on Result of subtraction Copyright Timo Autiokari, 1998-2006