Plane-based external camera calibration with accuracy measured by relative deflection angle Chunhui Cui ， KingNgiNgan Journal Image Communication Volume 25 Issue 3, March, 2010 Elsevier Science Inc. New York, NY, USA South Taiwan University ESLab Professor ： Tsai, Lian-Jou Student ： Tsai, Yu-Ming PPT Production rate ： 100% Date ： 2011/09/28

Outline Introduction Basic equations Proposed pair-wise pose estimation Homography estimation (R, t) estimation Accuracy measure based on relative deflection angle Experimental results Conclusion 1 South Taiwan University ESLab

Camera calibration has always been an essential component of photogrammetric measurement. Traditional Photogrammetric Calibration employs the reference grids and determines the intrinsic matrix K using images of a known object point array. External camera calibration ： estimate the poses of different cameras and register them to get her in a world coordinate system. The separation of the internal and external calibration benefits both the accuracy and efficiency of parameter estimation. 2 South Taiwan University ESLab Introduction (1/2)

3 South Taiwan University ESLab Introduction (2/2) In this paper, we present an efficient plane-based external calibration method to estimate the relative pose between two neighboring cameras. Based on homography, three different pair-wise estimation algorithms are proposed to recover the rotation and translation between cameras. It is convenient to take the RDA measure since it does not require any prior knowledge of the true 3D coordinates of control points.

4 South Taiwan University ESLab Camera model Pinhole camera model, the relationship between a 3D point M and its image projection m.

5 South Taiwan University ESLab Homography (1/3) Suppose two cameras capture a planar pattern 3D Point ： M Camera2 ： C2 Camera1 ： C1

choose C1 as the world coordinate system. 6 South Taiwan University ESLab Homography (2/3) m1 is the projection of M onto camera1 m2 is the projection of M onto camera2

where λ is an unknown arbitrary scalar. 7 South Taiwan University ESLab Homography (3/3)

8 South Taiwan University ESLab pair-wise pose estimation (1/6) We apply Zhang’s method to do the intrinsic calibration for each individual camera. Only the external calibration is necessary every time the cameras are moved and refocused to capture new 3D video. We only use the planar pattern to estimate the relative relationship between two neighboring cameras. It is argued that the chaining procedure is prone to errors. Accurate and stable pair-wise calibration can benefit the accuracy of transform chaining.

9 South Taiwan University ESLab pair-wise pose estimation (2/6) Using this information, (R, t) between two cameras can be determined through an arbitrarily chosen model plane. using these estimates may result in very unstable calibration results, since the accuracy of corner detection highly depends on the plane location and orientation.

10 South Taiwan University ESLab pair-wise pose estimation (3/6) Homography estimation Each image pair, one view from camera1 and the other from camera2, leads to a homography H. Suppose there are totally P image pairs and then we can estimate P homographies Hi (i=1,2,…,P) induced by different planes.

11 South Taiwan University ESLab pair-wise pose estimation (4/6) Calculation of n and λ P different stereo views lead to P different normals ni. The plane position and orientation (R1, t1) relative to C1, where the plane is now assumed on Z=0 of the world coordinate system.

12 South Taiwan University ESLab pair-wise pose estimation (5/6)

13 South Taiwan University ESLab pair-wise pose estimation (6/6) The collineation matrix G has median singular value equal to one. That matrix K1, K2 and H are known,so we can compute λ and then recover the matrix G.

14 South Taiwan University ESLab (R, t) estimation A linear to solve the rotation R and translation t. As each model plane introduces a normal ni, a homography Hi and thus a matrix Gi, by stacking P equations.

15 South Taiwan University ESLab Implicit orthogonal constraint (1/3) The three row vectors form an orthonormal basis of R^3.

16 South Taiwan University ESLab Implicit orthogonal constraint (2/3) One of the necessary conditions to guarantee the orthogonality of R.

17 South Taiwan University ESLab Implicit orthogonal constraint (3/3)

18 South Taiwan University ESLab Two-step method We obtain the uniform ratio o fthe three elements of t vector ： t vector is reduced to a single scale s as ： While keeps the problem linear. (R, t) estimated by this method not only conform to the homography geometry, but also satisfy the orthogonal constraint R^T ‧ R=1 much better.

19 South Taiwan University ESLab Three-step method (1/2) In three dimensions a rotation can be defined by a single angle of rotation θ, and the direction of a unit vector ： v is the corresponding eigenvector ； Rv = v (a) (c)

20 South Taiwan University ESLab Three-step method (2/2) Estimate cosθ, sinθ and vector t together by stacking P such equations ： Based on the above description, the Three-step method is outlined as follows ： (1) Use Two-step method to compute the initial estimation of t. (2) Estimate vector v based on (a) with t fixed. (3) Estimate parameters cosθ, sinθ and refine t together based on (b) with v fixed. cosθ, sinθ and v already computed, the rotation matrix R can be recovered by (c). (b)

21 South Taiwan University ESLab Nonlinear method We should impose the constraint R^T ‧ R=1 explicitly in the maximum likelihood estimation and it turns out to be a constrained nonlinear minimization problem.

22 South Taiwan University ESLab relative deflection angle (1/4) It can be achieved with good measures by: (1)Using lens with larger focal length so that the array of pixels focus on a smaller area of the scene. (2) Reducing the distance between the scene and the cameras to take a close look. (3) Directly increasing the image resolution. But the focal length and resolution of the digital camera will not benefit the measurement. We propose to measure the calibration accuracy based on the relative deflection angle(RDA).

23 South Taiwan University ESLab Therefore the deflection angle θerr integrates the errors from both internal and external calibration. The mean value E[θerr] over all the control points can be reasonably utilized to reveal the inaccuracy of stereo camera calibration. relative deflection angle (2/4)

24 South Taiwan University ESLab relative deflection angle (3/4) Consider a pixel rectangle of size du*dv. Uniform digitization noise in the rectangle has a standard deviation ： This digitization error corresponds to the uncertainty of the projection of a 3D point onto the image plane, which can be measured as the angle of the deflected projection rays.

25 South Taiwan University ESLab relative deflection angle (4/4) θsys is associated with the image resolution and the camera focal length.

26 South Taiwan University ESLab Experimental results Simulation on (R, t) estimation Two cameras capture a planar pattern with9*12 50mm square corners. Total 30 different plane poses are used for calibration. The second camera is translated and rotated by ； Add zero mean uncorrelated Gaussian noise to the image projections of the control points (square corners on the plane) from a standard deviation of 0.1 pixels up to 0.9 pixels. The relative estimation are used to measure the calibration accuracy,The results of 50 simulations are summarized in ：

27 South Taiwan University ESLab

28 South Taiwan University ESLab

29 South Taiwan University ESLab Test on RDA metric (1/4) For all three experiments, the nonlinear method initialized by 3-step is applied to estimate the external parameters. For comparison, we apply another error metric to measure the calibration accuracy. The mean square distance MSD defined in following will be used as the second error metric to measure the calibration results.

30 South Taiwan University ESLab Test on RDA metric (2/4) In the first experiment, the baseline of the two cameras is kept 1m fixed. We change the distance of the pattern from the cameras as well as the pattern orientations.

31 South Taiwan University ESLab Test on RDA metric (3/4) In the second experiment, the camera setup is varied in terms of the baseline length. the distance between the pattern and the baseline midpoint is slightly adjusted such that the distances from the pattern to the two cameras are the same and kept unchanged

32 South Taiwan University ESLab Test on RDA metric (4/4) we change the camera focal length while keeping the baseline length and the viewing distance fixed.

33 South Taiwan University ESLab Multi-camera calibration (1/2) The intrinsic parameters for the five cameras are estimated individually beforehand using Zhang’s method. A checkerboard pattern with 9*12 50 mm square corners is used for providing control points. Two different system setups are tested as sketched in Fig. 7.

34 South Taiwan University ESLab Multi-camera calibration (2/2) We apply the proposed RDA metric to measure the calibration errors. To investigate the error introduced by the chaining of pair-wise (R, t) estimation.

35 South Taiwan University ESLab Conclusion we present a convenient and efficient method to calibrate a typical multi-camera system. Three different algorithms are proposed based on homography. As the orthogonal constraint is fully imposed in the nonlinear method, it achieves the best calibration results in terms of the robustness to data noise. The Three-step and Nonlinear methods is more flexible and universal than the traditional multi-camera calibration methods, where all the cameras are required to capture the same control points. The RDA can be employed to provide a fair and meaningful evaluation on the calibration results for different cameras under different working conditions.

Thanks for your listening!! Any questions? South Taiwan University ESLab