1. Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction
2016 the first International Workshop on Pattern Recognition (IWPR 2016) Tokyo, Japan during May 11-13, Kwan Pang Tsuia, Kin Hong Wongb*, Changling Wanga, Ho Chuen Kamb, Hing Tuen Yaub, and Ying Kin Yu aDepartment of Mechanical Engineering, The Chinese University of Hong Kong, Hong Kong bDepartment of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong
Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

2. Faculty of Engineering, The Chinese University of Hong Kong (CUHK)
Electronic Engineering (since 1970)
Computer Science & Engineering (since 1973)
Information Engineering (since 1989)
Systems Engineering & Engineering Management (since 1991)
Mechanical and Automation Engineering (since 1994)
110 faculty members
2,200 undergraduates (15% non-local)
800 postgraduates
Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

3. The Chinese University of Hong Kong
Department of Computer Science and Engineering
Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

4. Overview Objectives Theory Implementation Results Conclusion
Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

5. Objectives
Investigate different methods to calibrate a 3-D scanning system consisting of multiple Kinect sensors
Reconstruct the full surface model of an object
Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

6. 3D Scanning Sensor Types
Laser
Very Accurate
Expensive ( up to US$50,000)
Bulky
LED
Accurate
Mid-range price ( > US$ 2000)
Infra-red
Microsoft Kinect, Structure Sensor
Cheap ( US$100~200)
Less accurate
Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

7. Motivations
3-D scanning is important in many scientific and industrial applications
3-D scanners are consumer grade products which are available at low cost
Previous work:
Single Kinect to move around the object to obtain result -- Relatively difficult to be handled and carried out perfectly
Iterative Closest Point (ICP)
Kinect Fusion
Target surface too smooth
Identifiable features too little
Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

8. Theory for method 1 Calibrate the pose between Kinect J and K first
Theory for method 1 Calibrate the pose between Kinect J and K first. Then K and L etc.
Kinect L
Kinect M
Move the sphere and take samples
Kinect J
Kinect K
Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

9. Theory for method 1 Overview of the system Method 1 :
Using Sphere as a calibration object (Staranowicz et. al. paper)
Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

10. Theory for method 2 Calibrate the pose between Kinect J and K first
Theory for method 2 Calibrate the pose between Kinect J and K first. Then K and L etc.
Kinect M
Kinect L
Move the checker board and take samples
Kinect J
Kinect K
Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

11. Theory for method 2 Method 2 : Checkerboard + Normal Equation
2 x Kinects capture same checkerboard image
Normal Equation
MB = A
MBBT = ABT
MBBT (BBT )−1 = ABT (BBT )−1
M = ABT (BBT )−1
Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

12. Theory for method 3 Move the cube and take samples
Method 3 : Checkerboard + Rotation Averaging
In theory all Kinects can be calibrated together at one go
Move the cube and take samples
Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

13. Rotation Averaging
Weiszfeld algorithm to find the average Rotation, modified from the paper by Hartley et. al.
Rjk : The relative rotation between Kinect j and Kinect k. It is to be found by our algorithm
Rj,i : The ith sample of the rotation between the checkerboard facing Kinect j and Kinect j, this is to be found by camera calibration or pose estimation methods.
Rk,i : The ith sample of the rotation between the checkerboard facing Kinect k and Kinect k, this is to be found by camera calibration or pose estimation methods.
R90 : Rotation of 90 degrees, that is the rotation between the checker board facing j and k, because they are two faces of a cube so the rotation should be 90 degrees.
Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

14. Take N images (e.g. N = 20) of the cube with different orientations, can find sample pairs of poses {Rj,i, Rk,i}, for i = 1, 2, .., N, using standard camera calibration or pose estimation methods.1 Using these rotations as inputs, the Weiszfeld algorithm8 is able to find the rotation Rj,k between Kinect j and Kinect k if it has enough samples of the rotation pairs, i.e. i = 1, 2, ..N for N ≥ 3
Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

15. Implementation Issue Infra-red Interference among Kinects
Microsoft Shake & Sense
Turn on & off the Kinect one by one
Connects multiple Kinects to a single PC machine
Kinect may occupy whole USB channel resources
Each USB PCI card control 1 x Kinect
Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

16. Result for method 1
Method 1: Sphere Approach(Staranowicz et. al. paper). Result is bad.
Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

17. Result for method 2 Method2: Checkerboard + Normal Equation Approach
Demo
demo_duck_VR.mp4 Or
Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

18. Simulation only, promising
Result for method 3.
Simulation only, promising
Checkerboard + Rotation Averaging Approach
The error angle ∆θ is the angle of the axis-angle representation of
∆Rerror , where ∆R error = RTjk_found ∗ Rjk_grouth_truth.
Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

19. Conclusion Sphere Approach
Consumed lot of CPU resources by 3D Sphere Hough
Poor Performance & Inaccurate
Checkerboard + Normal Equation
Accurately build full seamless 360 view 3D object model
Need to calibrate each pair of Kinect separately
Cube with checker pattern + Rotation Averaging
Performance is promising
Can potentially calibrate 4 Kinects at the same time
Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

20. Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

21. Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

22. Thanks
Q & A
Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a

23. END
Calibration of Multiple Kinect Depth Sensors for Full Surface Model Reconstruction (2016) v.3a