1. Vision, Video And Virtual Reality 3D Vision Lecture 13 Calibration CSC 59866CD Fall 2004 Zhigang Zhu, NAC 8/203A http://www-cs.engr.ccny.cuny.edu/~zhu/ Capstone2004/Capstone_Sequence2004.html

2. Vision, Video And Virtual Reality Lecture Outline n Calibration: Find the intrinsic and extrinsic parameters l Problem and assumptions l Direct parameter estimation approach l Projection matrix approach n Direct Parameter Estimation Approach l Basic equations (from Lecture 5) l Estimating the Image center using vanishing points l SVD (Singular Value Decomposition) and Homogeneous System l Focal length, Aspect ratio, and extrinsic parameters l Discussion: Why not do all the parameters together? n Projection Matrix Approach (…after-class reading) l Estimating the projection matrix M l Computing the camera parameters from M l Discussion n Comparison and Summary l Any difference?

3. Vision, Video And Virtual Reality Problem and Assumptions n Given one or more images of a calibration pattern, n Estimate l The intrinsic parameters l The extrinsic parameters, or l BOTH n Issues: Accuracy of Calibration l How to design and measure the calibration pattern n Distribution of the control points to assure stability of solution – not coplanar n Construction tolerance one or two order of magnitude smaller than the desired accuracy of calibration n e.g. 0.01 mm tolerance versus 0.1mm desired accuracy l How to extract the image correspondences n Corner detection? n Line fitting? l Algorithms for camera calibration given both 3D-2D pairs n Alternative approach: 3D from un-calibrated camera

4. Vision, Video And Virtual Reality Camera Model n Coordinate Systems l Frame coordinates (x im, y im ) pixels l Image coordinates (x,y) in mm l Camera coordinates (X,Y,Z) l World coordinates (X w,Y w,Z w ) n Camera Parameters l Intrinsic Parameters (of the camera and the frame grabber): link the frame coordinates of an image point with its corresponding camera coordinates l Extrinsic parameters: define the location and orientation of the camera coordinate system with respect to the world coordinate system ZwZw XwXw YwYw Y X Z x y O PwPw P p x im y im (x im,y im ) Pose / Camera Object / World Image frame Frame Grabber

5. Vision, Video And Virtual Reality Linear Version of Perspective Projection n World to Camera l Camera: P = (X,Y,Z) T l World: P w = (X w,Y w,Z w ) T l Transform: R, T n Camera to Image l Camera: P = (X,Y,Z) T l Image: p = (x,y) T l Not linear equations n Image to Frame l Neglecting distortion l Frame (x im, y im ) T n World to Frame l (X w,Y w,Z w ) T -> (x im, y im ) T l Effective focal lengths n f x = f/s x, f y =f/s y

6. Vision, Video And Virtual Reality Direct Parameter Method n Extrinsic Parameters l R, 3x3 rotation matrix Three angles  l T, 3-D translation vector n Intrinsic Parameters l fx, fy :effective focal length in pixel  = fx/fy = sy/sx, and fx l (ox, oy): known Image center -> (x,y) known l k 1, radial distortion coefficient: neglect it in the basic algorithm n Same Denominator in the two Equations l Known : (Xw,Yw,Zw) and its (x,y) l Unknown: r pq, Tx, Ty, fx, fy

7. Vision, Video And Virtual Reality Linear Equations n Linear Equation of 8 unknowns v = (v1,…,v8) Aspect ratio:  = fx/fy l Point pairs, {(Xi, Yi,, Zi) (xi, yi) } drop the ‘ and subscript “w”

8. Vision, Video And Virtual Reality Homogeneous System n Homogeneous System of N Linear Equations l Given N corresponding pairs {(Xi, Yi,, Zi) (xi, yi) }, i=1,2,…N l 8 unknowns v = (v1,…,v8) T, 7 independent parameters n The system has a nontrivial solution (up to a scale) l IF N >= 7 and N points are not coplanar => Rank (A) = 7 l Can be determined from the SVD of A

9. Vision, Video And Virtual Reality SVD: definition n Singular Value Decomposition: l Any mxn matrix can be written as the product of three matrices Singular values  i are fully determined by A D is diagonal: d ij =0 if i  j; dii =  i (i=1,2,…,n)  1   2  …   N  0 n Both U and V are not unique n Columns of each are mutual orthogonal vectors V1V1 U1U1 Appendix A.6

10. Vision, Video And Virtual Reality SVD: properties n 1. Singularity and Condition Number l nxn A is nonsingular IFF all singular values are nonzero l Condition number : degree of singularity of A n A is ill-conditioned if 1/C is comparable to the arithmetic precision of your machine; almost singular n 2. Rank of a square matrix A l Rank (A) = number of nonzero singular values n 3. Inverse of a square Matrix l If A is nonsingular l In general, the pseudo-inverse of A n 4. Eigenvalues and Eigenvectors (questions) Eigenvalues of both A T A and AA T are  i 2 (  i > 0) l The columns of U are the eigenvectors of AA T (mxm) l The columns of V are the eigenvectors of A T A (nxn)

11. Vision, Video And Virtual Reality SVD: Application 1 n Least Square l Solve a system of m equations for n unknowns x(m >= n) l A is a mxn matrix of the coefficients l b (  0) is the m-D vector of the data l Solution: l How to solve: compute the pseudo-inverse of A T A by SVD n (A T A) + is more likely to coincide with (A T A) -1 given m > n n Always a good idea to look at the condition number of A T A nxn matrixPseudo-inverse

12. Vision, Video And Virtual Reality SVD: Application 2 n Homogeneous System l m equations for n unknowns x(m >= n-1) l Rank (A) = n-1 (by looking at the SVD of A) l A non-trivial solution (up to a arbitrary scale) by SVD: l Simply proportional to the eigenvector corresponding to the only zero eigenvalue of A T A (nxn matrix) n Note: l All the other eigenvalues are positive because Rank (A)=n-1 l For a proof, see Textbook p. 324-325 In practice, the eigenvector (i.e. v n ) corresponding to the minimum eigenvalue of A T A, i.e.  n 2

13. Vision, Video And Virtual Reality SVD: Application 3 n Problem Statements l Numerical estimate of a matrix A whose entries are not independent l Errors introduced by noise alter the estimate to  n Enforcing Constraints by SVD l Take orthogonal matrix A as an example l Find the closest matrix to Â, which satisfies the constraints exactly n SVD of  n Observation: D = I (all the singular values are 1) if A is orthogonal n Solution: changing the singular values to those expected

14. Vision, Video And Virtual Reality Homogeneous System n Homogeneous System of N Linear Equations l Given N corresponding pairs {(Xi, Yi,, Zi) (xi, yi) }, i=1,2,…N l 8 unknowns v = (v1,…,v8) T, 7 independent parameters n The system has a nontrivial solution (up to a scale) l IF N >= 7 and N points are not coplanar => Rank (A) = 7 l Can be determined from the SVD of A l Rows of V T : eigenvectors {e i } of A T A Solution: the 8 th row e 8 corresponding to the only zero singular value 8 =0 n Practical Consideration l The errors in localizing image and world points may make the rank of A to be maximum (8) l In this case select the eigenvector corresponding to the smallest eigenvalue.

15. Vision, Video And Virtual Reality Scale Factor and Aspect Ratio Equations for scale factor  and aspect ratio  n Knowledge: R is an orthogonal matrix n Second row (i=j=2): n First row (i=j=1) v 1 v 2 v 3 v 4 v 5 v 6 v 7 v 8

16. Vision, Video And Virtual Reality Rotation R and Translation T Equations for first 2 rows of R and T given  and |  | n First 2 rows of R and T can be found up to a common sign s (+ or -) n The third row of the rotation matrix by vector product n Remaining Questions : l How to find the sign s? l Is R orthogonal? l How to find Tz and fx, fy?

17. Vision, Video And Virtual Reality Find the sign s Facts: l fx > 0 l Zc >0 l x known l Xw,Yw,Zw known Solution  Check the sign of Xc  Should be opposite to x ZwZw XwXw YwYw Y X Z x y O PwPw P p x im y im (x im,y im )

18. Vision, Video And Virtual Reality Rotation R : Orthogonality Question: l First 2 rows of R are calculated without using the mutual orthogonal constraint n Solution: l Use SVD of estimate R Replace the diagonal matrix D with the 3x3 identity matrix

19. Vision, Video And Virtual Reality Find Tz, Fx and Fy Solution l Solve the system of N linear equations with two unknown n Tx, fx l Least Square method l SVD method to find inverse a i2 a i1 bi

20. Vision, Video And Virtual Reality Direct parameter Calibration Summary n Algorithm (p130-131) 1. Measure N 3D coordinates (Xi, Yi,Zi) 2. Locate their corresponding image points (xi,yi) - Edge, Corner, Hough 3. Build matrix A of a homogeneous system Av = 0 4. Compute SVD of A, solution v 5. Determine aspect ratio  and scale |  | 6. Recover the first two rows of R and the first two components of T up to a sign 7. Determine sign s of  by checking the projection equation 8. Compute the 3 rd row of R by vector product, and enforce orthogonality constraint by SVD 9. Solve Tz and fx using Least Square and SVD, then fy = fx /  YwYw XwXw ZwZw

21. Vision, Video And Virtual Reality Discussions n Questions l Can we select an arbitrary image center for solving other parameters? l How to find the image center (ox,oy)? l How about to include the radial distortion? l Why not solve all the parameters once ? n How many unknown with ox, oy? --- 20 ??? – projection matrix method

22. Vision, Video And Virtual Reality Estimating the Image Center n Vanishing points: l Due to perspective, all parallel lines in 3D space appear to meet in a point on the image - the vanishing point, which is the common intersection of all the image lines

23. Vision, Video And Virtual Reality Estimating the Image Center n Vanishing points: l Due to perspective, all parallel lines in 3D space appear to meet in a point on the image - the vanishing point, which is the common intersection of all the image lines VP1

24. Vision, Video And Virtual Reality Estimating the Image Center n Vanishing points: l Due to perspective, all parallel lines in 3D space appear to meet in a point on the image - the vanishing point, which is the common intersection of all the image lines n Important property: Vector OV (from the center of projection to the vanishing point) is parallel to the parallel lines O VP1 Y X Z

25. Vision, Video And Virtual Reality Estimating the Image Center n Vanishing points: l Due to perspective, all parallel lines in 3D space appear to meet in a point on the image - the vanishing point, which is the common intersection of all the image lines VP1 VP2

26. Vision, Video And Virtual Reality Estimating the Image Center n Orthocenter Theorem: l Input: three mutually orthogonal sets of parallel lines in an image l T: a triangle on the image plane defined by the three vanishing points l Image center = orthocenter of triangle T l Orthocenter of a triangle is the common intersection of the three altitudes VP1 VP2 VP3

27. Vision, Video And Virtual Reality Estimating the Image Center n Orthocenter Theorem: l Input: three mutually orthogonal sets of parallel lines in an image l T: a triangle on the image plane defined by the three vanishing points l Image center = orthocenter of triangle T l Orthocenter of a triangle is the common intersection of the three altitudes VP1 VP2 VP3

28. Vision, Video And Virtual Reality Estimating the Image Center n Orthocenter Theorem: l Input: three mutually orthogonal sets of parallel lines in an image l T: a triangle on the image plane defined by the three vanishing points l Image center = orthocenter of triangle T l Orthocenter of a triangle is the common intersection of the three altitudes n Orthocenter Theorem: l WHY? VP1 VP2 VP3 h3h3 h1h1 h1h1 (o x,o y )

29. Vision, Video And Virtual Reality Estimating the Image Center n Assumptions: l Known aspect ratio l Without lens distortions n Questions: l Can we solve both aspect ratio and the image center? l How about with lens distortions? VP1 VP2 VP3 h3h3 h1h1 h1h1 (o x,o y ) ?

30. Vision, Video And Virtual Reality Direct parameter Calibration Summary n Algorithm (p130-131) 0. Estimate image center (and aspect ratio) 1. Measure N 3D coordinates (Xi, Yi,Zi) 2. Locate their corresponding image (xi,yi) - Edge, Corner, Hough 3. Build matrix A of a homogeneous system Av = 0 4. Compute SVD of A, solution v 5. Determine aspect ratio  and scale |  | 6. Recover the first two rows of R and the first two components of T up to a sign 7. Determine sign s of  by checking the projection equation 8. Compute the 3 rd row of R by vector product, and enforce orthogonality constraint by SVD 9. Solve Tz and fx using Least Square and SVD, then fy = fx /  YwYw XwXw ZwZw

31. Vision, Video And Virtual Reality Remaining Issues and Possible Solution n Original assumptions: l Without lens distortions l Known aspect ratio when estimating image center l Known image center when estimating others including aspect ratio n New Assumptions l Without lens distortion Aspect ratio is approximately 1, or  = fx/fy = 4:3 ; image center about (M/2, N/2) given a MxN image n Solution (?) 1. Using  = 1 to find image center (ox, oy) 2. Using the estimated center to find others including  3. Refine image center using new  ; if change still significant, go to step 2; otherwise stop l Projection Matrix Approach

32. Vision, Video And Virtual Reality Linear Matrix Equation of perspective projection n Projective Space l Add fourth coordinate n P w = (X w,Y w,Z w, 1 ) T l Define (u,v,w) T such that n u / w =x im, v / w =y im n 3x4 Matrix E ext l Only extrinsic parameters l World to camera n 3x3 Matrix E int l Only intrinsic parameters l Camera to frame n Simple Matrix Product! Projective Matrix M= M int M ext l (X w,Y w,Z w ) T -> (x im, y im ) T l Linear Transform from projective space to projective plane l M defined up to a scale factor – 11 independent entries

33. Vision, Video And Virtual Reality Projection Matrix M n World – Frame Transform l Drop “im” and “w” l N pairs (xi,yi) (Xi,Yi,Zi) l Linear equations of m n 3x4 Projection Matrix M l Both intrinsic (4) and extrinsic (6) – 10 parameters

34. Vision, Video And Virtual Reality Step 1: Estimation of projection matrix n World – Frame Transform l Drop “im” and “w” l N pairs (xi,yi) (Xi,Yi,Zi) n Linear equations of m l 2N equations, 11 independent variables l N >=6, SVD => m up to a unknown scale

35. Vision, Video And Virtual Reality Step 2: Computing camera parameters n 3x4 Projection Matrix M l Both intrinsic and extrinsic n From M^ to parameters (p134-135) Find scale |  by using unit vector R 3 T Determine T z and sign of  from m 34 (i.e. q 43 ) l Obtain R 3 T l Find (Ox, Oy) by dot products of Rows q1. q3, q2.q3, using the orthogonal constraints of R l Determine fx and fy from q1 and q2 (Eq. 6.19) Wrong???) l All the remainings: R 1 T, R 2 T, Tx, Ty l Enforce orthognoality on R?

36. Vision, Video And Virtual Reality Comparisons n Direct parameter method and Projection Matrix method n Properties in Common: l Linear system first, Parameter decomposition second l Results should be exactly the same n Differences l Number of variables in homogeneous systems n Matrix method: All parameters at once, 2N Equations of 12 variables n Direct method in three steps: N Equations of 8 variables, N equations of 2 Variables, Image Center – maybe more stable l Assumptions n Matrix method: simpler, and more general; sometime projection matrix is sufficient so no need for parameter decompostion n Direct method: Assume known image center in the first two steps, and known aspect ratio in estimating image center

37. Vision, Video And Virtual Reality Guidelines for Calibration n Pick up a well-known technique or a few n Design and construct calibration patterns (with known 3D) n Make sure what parameters you want to find for your camera n Run algorithms on ideal simulated data l You can either use the data of the real calibration pattern or using computer generated data l Define a virtual camera with known intrinsic and extrinsic parameters l Generate 2D points from the 3D data using the virtual camera l Run algorithms on the 2D-3D data set n Add noises in the simulated data to test the robustness n Run algorithms on the real data (images of calibration target) n If successful, you are all set n Otherwise: l Check how you select the distribution of control points l Check the accuracy in 3D and 2D localization l Check the robustness of your algorithms again l Develop your own algorithms  NEW METHODS?

38. Vision, Video And Virtual Reality Next n 3D reconstruction using two cameras Stereo Vision