Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction 2016 the first International Workshop on Pattern Recognition (IWPR 2016) Tokyo, Japan during May 11-13, 2016 Kwan Pang Tsuia, Kin Hong Wongb*, Changling Wanga, Ho Chuen Kamb, Hing Tuen Yaub, and Ying Kin Yu aDepartment of Mechanical Engineering, The Chinese University of Hong Kong, Hong Kong bDepartment of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

Faculty of Engineering, The Chinese University of Hong Kong (CUHK) Electronic Engineering (since 1970) Computer Science & Engineering (since 1973) Information Engineering (since 1989) Systems Engineering & Engineering Management (since 1991) Mechanical and Automation Engineering (since 1994) 110 faculty members 2,200 undergraduates (15% non-local) 800 postgraduates Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

The Chinese University of Hong Kong Department of Computer Science and Engineering Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

Overview Objectives Theory Implementation Results Conclusion Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

Objectives Investigate different methods to calibrate a 3-D scanning system consisting of multiple Kinect sensors Reconstruct the full surface model of an object Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

3D Scanning Sensor Types Laser Very Accurate Expensive ( up to US$50,000) Bulky LED Accurate Mid-range price ( > US$ 2000) Infra-red Microsoft Kinect, Structure Sensor Cheap ( US$100~200) Less accurate Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

Motivations 3-D scanning is important in many scientific and industrial applications 3-D scanners are consumer grade products which are available at low cost Previous work: Single Kinect to move around the object to obtain result -- Relatively difficult to be handled and carried out perfectly Iterative Closest Point (ICP) Kinect Fusion Target surface too smooth Identifiable features too little Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

Theory for method 1 Calibrate the pose between Kinect J and K first Theory for method 1 Calibrate the pose between Kinect J and K first. Then K and L etc. Kinect L Kinect M Move the sphere and take samples Kinect J Kinect K Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

Theory for method 1 Overview of the system Method 1 : Using Sphere as a calibration object (Staranowicz et. al. paper) Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

Theory for method 2 Calibrate the pose between Kinect J and K first Theory for method 2 Calibrate the pose between Kinect J and K first. Then K and L etc. Kinect M Kinect L Move the checker board and take samples Kinect J Kinect K Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

Theory for method 2 Method 2 : Checkerboard + Normal Equation 2 x Kinects capture same checkerboard image Normal Equation MB = A MBBT = ABT MBBT (BBT )−1 = ABT (BBT )−1 M = ABT (BBT )−1 Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

Theory for method 3 Move the cube and take samples Method 3 : Checkerboard + Rotation Averaging In theory all Kinects can be calibrated together at one go Move the cube and take samples Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

Rotation Averaging Weiszfeld algorithm to find the average Rotation, modified from the paper by Hartley et. al. Rjk : The relative rotation between Kinect j and Kinect k. It is to be found by our algorithm Rj,i : The ith sample of the rotation between the checkerboard facing Kinect j and Kinect j, this is to be found by camera calibration or pose estimation methods. Rk,i : The ith sample of the rotation between the checkerboard facing Kinect k and Kinect k, this is to be found by camera calibration or pose estimation methods. R90 : Rotation of 90 degrees, that is the rotation between the checker board facing j and k, because they are two faces of a cube so the rotation should be 90 degrees. Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

Take N images (e.g. N = 20) of the cube with different orientations, can find sample pairs of poses {Rj,i, Rk,i}, for i = 1, 2, .., N, using standard camera calibration or pose estimation methods.1 Using these rotations as inputs, the Weiszfeld algorithm8 is able to find the rotation Rj,k between Kinect j and Kinect k if it has enough samples of the rotation pairs, i.e. i = 1, 2, ..N for N ≥ 3 Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

Implementation Issue Infra-red Interference among Kinects Microsoft Shake & Sense Turn on & off the Kinect one by one Connects multiple Kinects to a single PC machine Kinect may occupy whole USB channel resources Each USB PCI card control 1 x Kinect Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

Result for method 1 Method 1: Sphere Approach(Staranowicz et. al. paper). Result is bad. Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

Result for method 2 Method2: Checkerboard + Normal Equation Approach Demo demo_duck_VR.mp4 Or https://www.youtube.com/watch?v=JVeMWvaJ6xc&feature=youtu.be Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

Simulation only, promising Result for method 3. Simulation only, promising Checkerboard + Rotation Averaging Approach The error angle ∆θ is the angle of the axis-angle representation of ∆Rerror , where ∆R error = RTjk_found ∗ Rjk_grouth_truth. Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

Conclusion Sphere Approach Consumed lot of CPU resources by 3D Sphere Hough Poor Performance & Inaccurate Checkerboard + Normal Equation Accurately build full seamless 360 view 3D object model Need to calibrate each pair of Kinect separately Cube with checker pattern + Rotation Averaging Performance is promising Can potentially calibrate 4 Kinects at the same time Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

Thanks Q & A Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

END Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a