1. 3D full object reconstruction from kinect Yoni Choukroun Elie Semmel Advisor: Yonathan Afflalo

2. Agenda  Project’s Goals  Project Development  Kinect  Registration  Reconstruction  Performances  Demonstration

3. Project’s Goals  Understand Kinect working  C++ development  Rigid Iterative closest point implementation  Unknown libraries integration on Windows Platform  Deal with Reconstruction problems

4. Development of the project  Papers read  Kinect depth maps acquisition  Simple rigid ICP implementation  Body reconstruction

5. Papers The model Project 3D SELF-PORTRAITS Hao Li, Etienne Vouga, Anton Gudym, Jonathan T. Barron, Linjie Luo, Gleb Gusev A Method for Registration of 3-D Shapes Besl, Paul J.; N.D. McKay (1992). Efficient Variants of the ICP Algorithm Szymon Rusinkiewicz, Marc Levoy

6. The model Project 3D SELF-PORTRAITS Hao Li, Etienne Vouga, Anton Gudym, Jonathan T. Barron, Linjie Luo, Gleb Gusev  Full body reconstruction in 3-D with multiples scan frames of the Kinect  Pipelined reconstruction  First pipeline station: scanning and fusion  Rigid ICP of all the scans to get the entire body  Poisson reconstruction for a better quality

8. Kinect

9. Some facts:  The Kinect moves from [+27 ; -27 ]  Our Kinect motor is controlled by an OPENNI 1.5 code.  The Kinect provides a 480X640 pixels resolution depth map.  The depth map returned by the Kinect is of a poor quality.

10. Kinect Intrinsic Matrix Calibration  application requires maximum accuracy rigorous calibration.  different users with different Kinect average calibration with an average intrinsic matrix calibration. Pros / Cons:  More precise calibration will improve the performances and the final result but it asks from the user a tiresome calibration step.

11. Registration Problem Given: Two shapes P and Q which partially overlap. Goal : Using only rigid transforms, register Q against P by minimizing the squared distance between them.

12.  Simple rigid ICP theory  Mathematical preliminaries  Calculation tools  Algorithm  Performances  Optimization A Method for Registration of 3-D Shapes Besl, Paul J.; N.D. McKay (1992).

13. ICP Problematic  Given set of corresponding points it is easy to find the transformation which stitch the two images together.  Here, we do not have any previous prior about the shapes (maybe just good initial conditions enough).  We need then to find the best correlation between points:  ITERATIVE CLOSEST POINT!

14. Iteratively find the pairs of closest points Solve: Correspondence Space P Q

15. A Method for Registration of 3-D Shapes Besl, Paul J.; N.D. McKay (1992).  Sampling (~1:30) the cloud (sparse matrix array of coordinates)

16. A Method for Registration of 3-D Shapes Besl, Paul J.; N.D. McKay (1992).  Euclidean Metric used to find the Closest point

17. Transformation Calculus:  Find rotation and translation:  Apply registration until threshold is reached.

18. ICP Algorithm Init the error to ∞ Compute correspondences Compute alignment Apply alignment Update error If error > threshold Y = CP(P,Q) (Rot,Trans,d) Q`= Rot(Q)+Trans

19. Efficient Variants of the ICP Algorithm Szymon Rusinkiewicz, Marc Levoy  Rigid ICP variant performances  Selection of points  Matching points  Weighting pairs  Rejecting pairs

20. Selection of points Simple variants give good enough results: we implemented both random and uniform sampling

21. Matching points  Simple implementation for good enough results (suppose to be small Kinect angles): we chose the Closest Point for pair corresponding

22. Reconstruction Methods :  Perform ICP between a frame and the result of the all precedent frames bond together  bad results because of noise accumulation.  Improvement: Perform ICP between pair of frames and apply transitivity to get the registrations to the first frame: Fn, the n-th frame Tr, the transformation between Fn+1 and Fn Rn and Tn, the corresponding Rotation and Translation.

23. Project Algorithm Kinect Depth maps acquisition OpenNi interface OpenCV conversion Filtering Registration Displaying After #scans registrations Body reconstruction

24. ICP Performances  Bad performances for big moves  Not too big rotations required  Number of iterations: 1-3 for small registrations to about 10 for big moves (in general converges to wrong local minimum )  Noisy depth map  Bad Kinect resolution affects image’s acquisition: holes in the target shape.  For complexity purpose, search the closest point only in small radius near the point: finding closest point complexity

25. Performances Good fusion result vs time  Tradeoff: Fusion result Vs Execution time  Number of sampling between 8000 and 12000 (static memory allocation)  Time for ICP running : 1 ~ 3 min ( depended on number of samples)  Scanning and reconstructing time less than one minute.  Total time between 1.2 mn to 3mn

26. Fusion Results  No pre/post processing used (noise filtering etc.)  Conversion to MeshLab and Matlab files for displaying (and for post processing like texturing…)  Matlab Implementation for triangulation.(Not used because of poor quality: points cloud looks better)

27. Demonstration https://www.dropbox.com/s/ep2r8ajsodmhw7v/My%20Movie.mp4?dl=0

28. References " 3D SELF-PORTRAITS " Hao Li, Etienne Vouga, Anton Gudym, Jonathan T. Barron, Linjie Luo, Gleb Gusev ACM Transactions on Graphics, Proceedings of the 6th ACM SIGGRAPH Conference and Exhibition in Asia 2013, 11/2013 – SIGGRAPH ASIA 2013 "A Method for Registration of 3-D Shapes" Besl, Paul J.; N.D. McKay (1992). IEEE Trans. on Pattern Analysis and Machine Intelligence (Los Alamitos, CA, USA: IEEE Computer Society) "Efficient Variants of the ICP Algorithm" Szymon Rusinkiewicz, Marc Levoy Stanford University