Kinect calibration Ilya Afanasyev Facoltà di Ingegneria Trento, /20 25/01/2012

Content 2/20 1. The description. 2. About the existing software. 3. Tests of Nicolas Burrus software binaries. 4. Open Kinect drivers installation. 5. Compilation of Nicolas Burrus source code. 6. Results. 7. Conclusions 8. References. 25/01/2012

The description. 3/20 The software for Kinect Calibration should be used for: 1. Saving the RGB images and IR-Depth data from Kinect sensor under OS Windows. 2. Calibration the Kinect sensor with obtaining [1]: - the intrinsic parameters (i.e. matrixes of intrinsic parameters for IR camera (K_ir) & RGB camera (K_rgb)). - the extrinsic parameters (i.e. extrinsic mapping between IR (depth) and RGB Kinect camera (R,T)). - distortion coefficients (kc_ir, kc_rgb). - the intrinsic and extrinsic parameters (i.e. matrixes of intrinsic parameters for IR camera (K_ir) & RGB camera (K_rgb) and extrinsic mapping between IR (depth) and RGB Kinect camera (R,T)). 3. Developing the interface of different algorithm with Kinect for different real-time object recognition applications. 16/01/2012

About the existing software. 4/20 The existing analogues of the software: Microsoft Kinect SDK 1. - The author doesn’t know analogues under Windows7/64 bit. It can be created, basing on Microsoft Kinect SDK for developers [6] under the platform.NET Framework ver There is the software developed by Nicolas Burrus [2] under OS Linux / Ubuntu / 32 bit, which is claimed to be compiled also with Windows-32 bit. Can it be used under Windows7/64 bit ?? -> It seems that NO!! 25/01/2012

Microsoft Kinect SDK for developers Microsoft Kinect SDK for developers [6] 5/20 1. SDK (Software Development Kids) includes drivers, APIs for raw sensor streams and human motion tracking, installation documents, and resource materials. It provides capabilities offered by the Microsoft Kinect device connected to computers under Windows 7 / Windows 8 to developers who build applications with C++, C#, or Visual Basic by using Microsoft Visual Studio /01/2012

Nicolas Burrus software [2] 6/20 Nicolas Burrus software [2]: There are different versions of the Burrus software: 1. Test programs running from binaries -> Windows binaries -> Kinect RGBDemo v > supports OpenNI/Nite backends [3] and has the experimental infrared support with OpenNI (still buggy) -> I tried to check this work for OpenNI backend under Windows 64bit -> there is no the extrinsic mapping between IR and RGB camera at Kinect calibration. 2. Compiling from source -> 2.1 Compilation on Windows ->  with OpenNI/Nite backends doesn’t give the extrinsic mapping between IR and RGB camera at Kinect calibration.  with Libfreenect backend -> I wasn’t succeed to compile under Windows 7/64 bit?! 2.2 Compilation on Linux (Ubuntu bit) -> supports Libfreenect backend -> the only workable variant with getting all calibration parameters (intrinsic, extrinsic and distortion) was realized on Alberto Fornaser’s PC (UniTN) 16/01/2012

7/20 1. OpenNI backend: 1. Nicolas Burrus software [1]: Kinect RGBDemo v > supports OpenNI/Nite backends [2] and has the experimental infrared support with OpenNI (still buggy) -> I tried to check this work for OpenNI backend under Windows 64bit. 2. Install Sensor Kinect drivers under Windows 32 bit [4] and OpenNI/Nite modules (Win32) [3] according to the instruction [5] and verified that they work properly. 3. Use RGB-D Capture [1] to grab Kinect RGB and IR images with intensity.raw and depth.raw files. -> the software doesn’t grab IR images!! -> only RGB image, intensity.raw, depth.raw and calibration file (calibration.yml without distortion parameters)!! 4. “calibrate-openni-intrinsics --pattern-size grab1 calibration.yml” -> gives “openni_calibration.yml” with Intrinsic matrix and distortion coefficients for Kinect RGB camera !! 5. “calibrate-openni-depth.exe --pattern-size grab1” gives partly processed figures with a message about errors. 6. The software doesn’t produce a file with xyz- coordinates and RGB-colors info of every point!!! 1 st attempt 16/01/2012 The using of Nicolas Burrus software [2] with OpenNI backend under Windows

The comparison of Kinect calibration results for - Libfreenect backend (under OS Linux / Ubuntu bit) - OpenNI backend (under OS Windows 32bit) - The same figures of Intrinsic Matrixes for Kinect RGB and Depth Cameras Libfreenect backendOpenNI backend The Depth Camera Calibration for OpenNI backend is NOT available!!! 16/01/2012

The comparison of Kinect calibration results for - Libfreenect backend (under OS Linux / Ubuntu bit) - OpenNI backend (under OS Windows 32bit) - 23/11/2011 There is NO extrinsic mapping between Kinect Depth and RGB cameras for OpenNI backend Libfreenect backendOpenNI backend The reprojection of 3D points from Kinect Depth Camera to RGB image for OpenNI is NOT possible!!

Install OpenKinect for Windows 7 Install OpenKinect for Windows 7 [7] 10/20 It is necessary to have: - Visual Studio 2010 (as a compiler on a 32 bit Windows 7 installation). - Kinect Sensor drivers for Windows 7 (OpenKinect-libfreenect ofTheo/ofxKinect) [9]. - freenect windows dependencies [8]:  libusb-win (Currently there is a breaking change in libusb , which I used!!);  pthreads-win32  Glut 17/01/2012

Kinect Sensor drivers installation Kinect Sensor drivers installation [7] 11/20 - Download & extract the latest GitHub repository [9]. - Make sure your Kinect sensor is plugged into a USB port on your computer. - Open Device Manager (Start  Control Panel  System and Security  System  Device Manager) - Find “Human Interface Device” (Xbox NUI Motor)  Update Driver Software  uninstall the previous Kinect drivers (if there are)  set path like C:\Special\OpenKinect_libfreenect_v0_1_1_56\pl atform\windows\inf\xbox nui camera\  the drivers will correspond to the processor type. 18/01/2012

Kinect Sensor drivers installation Kinect Sensor drivers installation [7] 12/20 19/01/2012 Check the files: libusb0.dll (x64) in C:\Windows\System32\ and libusb0.sys in C:\Windows\System32\drivers\ and libusb0.dll (x86) in C:\Windows\SysWOW64\

Cmake configuration & Generation of a Visual Studio 10 solution Cmake configuration & Generation of a Visual Studio 10 solution [7] 13/20 1. Open Cmake GUI and set the Browse Source path to /libfreenect/openkinect. 2. Set the browse build path to /libfreenect/openkinect/build 19/01/2012 The software should be configured when Kinect turned off!!

Cmake configuration & Generation of a Visual Studio 10 solution Cmake configuration & Generation of a Visual Studio 10 solution [7] 14/20 You will now need to point certain build paths to the freenect dependencies (pthread). 19/01/2012 The software should be configured when Kinect turned off!!

Cmake configuration & Generation of a Visual Studio 10 solution Cmake configuration & Generation of a Visual Studio 10 solution [7] 15/20 You will now need to point certain build paths to the freenect dependencies (libusb). 19/01/2012 If take "msvc" instead of "msvc_x64" -> 1 fatal error of compilation less !!! It doesn't matter if choose THREADS\LIB\ pthreadVC2.lib or pthreadVSE2.lib

Cmake configuration & Generation of a Visual Studio 10 solution Cmake configuration & Generation of a Visual Studio 10 solution [7] 16/20 19/01/2012 Run "Configure" and it should successfully complete. Click Generate afterwards and then open the libfreenect.sln file from the build folder the output was placed in - /openkinect/build/ Run Configure again. It should successfully complete the process this time but we still need to add the Glut references.

Visual Studio 10 solution Visual Studio 10 solution [7] 17/20 19/01/2012 RESULT: Rebuild All: 13 succeeded, 2 failed, 2 skipped. You will receive a few errors when building the solution but it should compile correctly regardless. The binaries will be placed in build/bin/Debug. Disable the Antivirus if it disturbs the creation of the VS10 solution! Choose Visual C and then select Build Solution from the Debug Menu or press F7.

The result of building the c for Windows 7 The result of building the c for Windows 7 [7] 18/20 The final step is to place a few.dll files in the appropriate folder and then we can run some of the samples provided:  Copy freenect.dll and freenect_sync.dll from \build\lib\Debug  Copy from dependencies folders: pthreadVC2.dll, glut32.dll, libusb0.dll (choose the correct version x86/x64)  Copy tiltdemo.exe, regtest.exe, glview.exe, glpclview.exe, glpclview.exe, regview.exe, hiview.exe from \build\bin\Debug  It can be helpful to copy freenect_sync.lib, freenect.lib, freenect.exp from \build\lib\Debug to \lib\ Successful start of demo-samples  Kinect works! 19/01/2012

Demo-samples of OpenKinect for Windows 7 19/20  tiltdemo.exe drives the Kinect motor.  regview.exe shows the combined IR&RGB video (IR up to 1 m)  regtest.exe grabs bad-quality pictures: -- registration_test_depth_registered.pgm, -- registration_test_rgb.ppm, -- registration_test_depth_raw.pgm, -- registration_test_depth_mm.pgm  glview.exe gives IR&RGB video in separate windows (IR up to 3.5 m) 19/01/2012

Demo-samples of OpenKinect for Windows 7 20/20  hiview.exe gives IR&RGB video in separate windows (IR up to 3.5 m, RGB has hi-quality & hi resolution)  glpclview.exe gives color 3D video with big shadows behind the objects.  cppview.exe produces a continuous list with messages under the phrase: “thread execution started” 19/01/2012

Compiling RGBDemo under Min GW from source with OpenKinect on Windows 7 [2] 21/20 The step-by-step procedure for Min GW, in case using libfreenect: 1. Install QT opensource for Windows. This will also install Min GW. 2. Add C:\Special\QtSDK\mingw\bin\ to the Path environment variable: Start  Computer (properties)  Advanced System Settings  Environmental Variables (button)  Variable: PATH  Edit  Add. 3. Install and run cmake on rgbdemo  Disable the NESTK_USE_OPENNI cmake variable  “Could not find module FindOpenCV.cmake”  “Configuring incomplete, errors occurred!” 4. Ungrouped Entries  OpenCV_DIR (NOT FOUND)  C:/Special/opencv.  QT_QMAKE_EXECUTABLE (NOT FOUND)  C:\Special\QtSDK\Desktop\Qt\4.7.4\mingw\bin\qmake.exe  it is better to use qmake.exe from QtSDK\mingw ?? 19/01/ nd attempt

Compiling RGBDemo under Min GW from source with OpenKinect on Windows 7 [2] 22/20 The step-by-step procedure for Min GW, in case using libfreenect:  Open the CMakeLists.txt in Qt Creator (from C:\Special\RGBDemo Source\RGBDemo Source\CMakeLists.txt open with C:\Special\QtSDK\QtCreator\bin\qtcreator.exe ).  It creates the folder: qtcreator-build  Add QT-qmake & OpenCV path to the Path environment variable: C:\Special\QtSDK\Desktop\Qt\4.7.4\mingw\bin\ C:\Special\opencv\ to the Path environment variable: Start  Computer (properties)  Advanced System Settings  Environmental Variables (button)  Variable: PATH  Edit  Add. 20/01/2012

Compiling RGBDemo under Min GW / MSVS 2010 from source with OpenKinect on Windows 7 [2] 23/20 The step-by-step procedure for Min GW / MSVS2010, in case using libfreenect:  Cmake  Ungrouped Entries  PKG_CONFIG_EXECUTABLE  C:\Special\opencv\OpenCVFindPkgConfig.cmake Results: Cmake can’t compile the QT & OpenCV_DIR  OpenCVConfig.cmake  We should use Qt!!  Qt  Run Cmake  “ CMAKE_MODULE_PATH C:\Special\qtcreator-build\ ”  Run Cmake  “ CUDA_TOOLKIT_ROOT_DIR not found or specified ” [10]  if set path to C:\Special\PCL_1_4_0\ which contains CUDA libraries, then we have the messages “ CMake Error: The source directory "C:/Special/PCL_1_4_0/3rdParty/flann/bin" does not appear to contain CMakeLists.txt. Specify --help for usage, or press the help button on the CMake GUI ”. 20/01/2012 Burrus software “ Burrus software “RGBDemo-0.6.1” is not installed on Window7-64bit!!

Conclusions 24/ Nicolas Burrus software with OpenNI binaries [2] works under Windows 7-64 bit, but the software doesn’t give the extrinsic mapping between Kinect Depth and RGB cameras and doesn’t produce a file with xyz- coordinates and RGB- colors info of every point!!! 2. Although the drivers of OpenKinect has been successfully installed under Windows 7-64 bit, the Nicolas Burrus source code RGBDemo (with LGPL License) for Kinect Calibration [2] is not compiled properly under Windows 7-64 bit.  It should be used another software or another Kinect drivers!! 3. Both Microsoft Kinect SDK for developers [6] and OpenKinect drivers [9] can be applicable to save RGB images and IR-Depth data to calibrate Kinect sensor and use it under Windows 7-64 bit, but no suitable software has not been produced yet. 25/01/2012

5.References 1. Camera Calibration and 3d Reconstruction. // OpenCV (Open Source Computer Vision) v2.1 documentation: bration_and_3d_reconstruction.html 2. Nicolas Burrus. Kinect. RGBDemo, calibrate and visualize Kinect output. // OpenNI Modules. // 4. SensorKinect drivers. // github social coding How-to: Successfully install Kinect on Windows (OpenNI and NITE). // Vangos Pterneas blog: successfully-install-kinect-windows-openni-nite.aspx 6. Microsoft Kinect SDK for developers. 25/20 16/01/2012

5.References 7. Install OpenKinect for Windows 7 and XP. // openkinect-windows-7-and-xp 8. Getting Started using OpenKinect. // “OpenCV: CUDA_TOOLKIT_ROOT_DIR not found or specified“. cuda_toolkit_root_dir-not-found-or-specified/ 26/20 16/01/2012