1. 3D Geometry and Camera Calibration

2. 3D Coordinate Systems Right-handed vs. left-handedRight-handed vs. left-handed x yz x yz

3. 2D Coordinate Systems y axis up vs. y axis down y axis up vs. y axis down Origin at center vs. cornerOrigin at center vs. corner Will often write ( u, v ) for image coordinatesWill often write ( u, v ) for image coordinates u v u u v v

4. 3D Geometry Basics 3D points = column vectors3D points = column vectors Transformations = pre-multiplied matricesTransformations = pre-multiplied matrices

5. Rotation Rotation about the z axisRotation about the z axis Rotation about x, y axes similar (cyclically permute x, y, z )Rotation about x, y axes similar (cyclically permute x, y, z )

6. Arbitrary Rotation Any rotation is a composition of rotations about x, y, and zAny rotation is a composition of rotations about x, y, and z Composition of transformations = matrix multiplication (watch the order!)Composition of transformations = matrix multiplication (watch the order!) Result: orthonormal matrixResult: orthonormal matrix – Each row, column has unit length – Dot product of rows or columns = 0 – Inverse of matrix = transpose

7. Arbitrary Rotation Rotate around x, y, then z :Rotate around x, y, then z : Don’t do this! Compute simple matrices and multiply them!Don’t do this! Compute simple matrices and multiply them!

8. Scale Scale in x, y, z :Scale in x, y, z :

9. Shear Shear parallel to xy plane:Shear parallel to xy plane:

10. Translation Can translation be represented by multiplying by a 3  3 matrix?Can translation be represented by multiplying by a 3  3 matrix? No.No. Proof:Proof:

11. Homogeneous Coordinates Add a fourth dimension to each point:Add a fourth dimension to each point: To get “real” (3D) coordinates, divide by w :To get “real” (3D) coordinates, divide by w :

12. Translation in Homogeneous Coordinates After divide by w, this is just a translation by ( t x, t y, t z )After divide by w, this is just a translation by ( t x, t y, t z )

13. Perspective Projection What does 4 th row of matrix do?What does 4 th row of matrix do? After divide,After divide,

14. Perspective Projection This is projection onto the z =1 planeThis is projection onto the z =1 plane Add scaling, etc.  pinhole camera modelAdd scaling, etc.  pinhole camera model (x,y,z)(x,y,z)(x,y,z)(x,y,z) ( x / z, y / z,1) (0,0,0) z =1

15. Putting It All Together: A Camera Model 3D point (homogeneous coords) Camera location Camera orientation Perspectiveprojection Scale to pixel size Translate to image center Then perform homogeneous divide, and get (u,v) coords

16. Putting It All Together: A Camera Model Extrinsics Intrinsics

17. World coordinates Camera coordinates Normalized device coordinates Eye coordinates Image coordinates Pixel coordinates

18. More General Camera Model Multiply all these matrices togetherMultiply all these matrices together Don’t care about “ z ” after transformationDon’t care about “ z ” after transformation Scale ambiguity  11 free parametersScale ambiguity  11 free parameters

19. Radial Distortion Radial distortion can not be represented by matrixRadial distortion can not be represented by matrix ( c u, c v ) is image center, u * img = u img – c u, v * img = v img – c v,  is first-order radial distortion coefficient( c u, c v ) is image center, u * img = u img – c u, v * img = v img – c v,  is first-order radial distortion coefficient

20. Camera Calibration Determining values for camera parametersDetermining values for camera parameters Necessary for any algorithm that requires 3D  2D mappingNecessary for any algorithm that requires 3D  2D mapping Method used depends on:Method used depends on: – What data is available – Intrinsics only vs. extrinsics only vs. both – Form of camera model

21. Camera Calibration – Example 1 Given:Given: – 3D  2D correspondences – General perspective camera model (11-parameter, no radial distortion) Write equations:Write equations:

22. Camera Calibration – Example 1 Linear equationLinear equation Overconstrained (more equations than unknowns)Overconstrained (more equations than unknowns) Underconstrained (rank deficient matrix – any multiple of a solution, including 0, is also a solution)Underconstrained (rank deficient matrix – any multiple of a solution, including 0, is also a solution)

23. Camera Calibration – Example 1 Standard linear least squares methods for Ax=0 will give the solution x=0Standard linear least squares methods for Ax=0 will give the solution x=0 Instead, look for a solution with |x|= 1Instead, look for a solution with |x|= 1 That is, minimize |Ax| 2 subject to |x| 2 =1That is, minimize |Ax| 2 subject to |x| 2 =1

24. Camera Calibration – Example 1 Minimize |Ax| 2 subject to |x| 2 =1Minimize |Ax| 2 subject to |x| 2 =1 |Ax| 2 = (Ax) T (Ax) = (x T A T )(Ax) = x T (A T A)x|Ax| 2 = (Ax) T (Ax) = (x T A T )(Ax) = x T (A T A)x Expand x in terms of eigenvectors of A T A: x =  1 e 1 +  2 e 2 +… x T (A T A)x = 1  1 2 + 2  2 2 +… |x| 2 =  1 2 +  2 2 +…Expand x in terms of eigenvectors of A T A: x =  1 e 1 +  2 e 2 +… x T (A T A)x = 1  1 2 + 2  2 2 +… |x| 2 =  1 2 +  2 2 +…

25. Camera Calibration – Example 1 To minimize 1  1 2 + 2  2 2 +… subject to  1 2 +  2 2 +… = 1 set  min = 1 and all other  i =0To minimize 1  1 2 + 2  2 2 +… subject to  1 2 +  2 2 +… = 1 set  min = 1 and all other  i =0 Thus, least squares solution is minimum (non- zero) eigenvalue of A T AThus, least squares solution is minimum (non- zero) eigenvalue of A T A

26. Camera Calibration – Example 2 Incorporating additional constraints into camera modelIncorporating additional constraints into camera model – No shear, no scale (rigid-body motion) – Square pixels – etc. These impose nonlinear constraints on camera parametersThese impose nonlinear constraints on camera parameters

27. Camera Calibration – Example 2 Option 1: solve for general perspective model, then find closest solution that satisfies constraintsOption 1: solve for general perspective model, then find closest solution that satisfies constraints Option 2: nonlinear least squaresOption 2: nonlinear least squares – Usually “gradient descent” techniques – Common implementations available (e.g. Matlab optimization toolbox)

28. Camera Calibration – Example 3 Incorporating radial distortionIncorporating radial distortion Option 1:Option 1: – Find distortion first (straight lines in calibration target) – Warp image to eliminate distortion – Run (simpler) perspective calibration Option 2: nonlinear least squaresOption 2: nonlinear least squares

29. Camera Calibration – Example 4 What if 3D points are not known?What if 3D points are not known? Structure from motion problem!Structure from motion problem! As we saw last time, can often be solved since # of knowns > # of unknownsAs we saw last time, can often be solved since # of knowns > # of unknowns

30. Multi-Camera Geometry Epipolar geometry – relationship between observed positions of points in multiple camerasEpipolar geometry – relationship between observed positions of points in multiple cameras Assume:Assume: – 2 cameras – Known intrinsics and extrinsics

31. Epipolar Geometry P C1C1C1C1 C2C2C2C2 p2p2p2p2 p1p1p1p1

32. P C1C1C1C1 C2C2C2C2 p2p2p2p2 p1p1p1p1 l2l2l2l2

33. P C1C1C1C1 C2C2C2C2 p2p2p2p2 p1p1p1p1 l2l2l2l2 Epipolar line Epipoles

34. Epipolar Geometry Goal: derive equation for l 2Goal: derive equation for l 2 Observation: P, C 1, C 2 determine a planeObservation: P, C 1, C 2 determine a plane P C1C1C1C1 C2C2C2C2 p2p2p2p2 p1p1p1p1 l2l2l2l2

35. Epipolar Geometry Work in coordinate frame of C 1Work in coordinate frame of C 1 Normal of plane is T  Rp 2, where T is relative translation, R is relative rotationNormal of plane is T  Rp 2, where T is relative translation, R is relative rotation P C1C1C1C1 C2C2C2C2 p2p2p2p2 p1p1p1p1 l2l2l2l2

36. Epipolar Geometry p 1 is perpendicular to this normal: p 1  (T  Rp 2 ) = 0p 1 is perpendicular to this normal: p 1  (T  Rp 2 ) = 0 P C1C1C1C1 C2C2C2C2 p2p2p2p2 p1p1p1p1 l2l2l2l2

37. Epipolar Geometry Write cross product as matrix multiplicationWrite cross product as matrix multiplication P C1C1C1C1 C2C2C2C2 p2p2p2p2 p1p1p1p1 l2l2l2l2

38. Epipolar Geometry p 1  T * R p 2 = 0  p 1 T E p 2 = 0p 1  T * R p 2 = 0  p 1 T E p 2 = 0 E is the essential matrixE is the essential matrix P C1C1C1C1 C2C2C2C2 p2p2p2p2 p1p1p1p1 l2l2l2l2

39. Essential Matrix E depends only on camera geometryE depends only on camera geometry Given E, can derive equation for line l 2Given E, can derive equation for line l 2 P C1C1C1C1 C2C2C2C2 p2p2p2p2 p1p1p1p1 l2l2l2l2

40. Fundamental Matrix Can define fundamental matrix F analogously, operating on pixel coordinates instead of camera coordinates u 1 T F u 2 = 0Can define fundamental matrix F analogously, operating on pixel coordinates instead of camera coordinates u 1 T F u 2 = 0 Advantage: can sometimes estimate F without knowing camera calibrationAdvantage: can sometimes estimate F without knowing camera calibration