1. EEC-693/793 Applied Computer Vision with Depth Cameras
Lecture 2
Wenbing Zhao 1

2. Outline Components of Kinect sensor Kinect for Windows SDK
Building first Kinect application 2

3. Components of Kinect for Windows
Color camera
Infrared (IR) emitter
IR depth sensor
Tilt motor
Microphone array
LED 3

4. The Color Camera Red, blue, green color images (RGB) Frame rate:
30 FPS at 640x480
12 FPS at 1280x960
Viewing range:
43 degrees vertical by 57 degrees horizontal 4

5. IR Emitter & IR Depth Sensor
IR emitter emits IR light in a pseudo-random dot pattern
Dotted light reflects off different objects
IR depth sensor reads reflected light => depth info
Depth stream resolution
640x480
320x240
80x60 5

6. Depth Data Processing 6

7. Tilt Motor
Used to change the camera and sensor’s angles to get the correct position of the human skeleton
Can tilt vertically (upwards/downwards) up to 27 degrees 7

8. Microphone Array
4 different microphones: locate the direction of the audio wave, not just capture the sound
Enhanced noise suppression, echo cancellation, beam-forming 8

9. LED Placed between camera and IR emitter
Used for indicating the status of the Kinect
Green indicates Kinect drivers loaded properly 9

10. Kinect for Xbox vs. Kinect for Windows
Near mode: as close as 40cm away from Kinect
Can be used for commercial applications
Can be used in a Win7 virtual machine: HyperV, VMWare, Parallels
For both
Track motion up to 12 feet (4 meters) away
Identical resolution 10

11. Kinect for Windows SDK Supported operating systems
Windows 7, Windows 8, Windows Embedded 7
Hardware requirements
Dual core 2.66 GHz or faster CPU
Dedicated USB 2.0 bus
2GB or higher RAM
Software requirement
Microsoft Visual Studio 2010 or higher
Kinect for Windows SDK: current version 1.8 11

12. Kinect for Windows SDK 12

13. Kinect for Windows SDK 13

14. How Applications Interact with Kinect14

15. Classification of Kinect SDK APIs15

16. Kinect Driver
The Kinect driver controls the camera, depth sensor, audio microphone array, and motion
Data passes between the sensor and the app in the form of data streams
Color data stream
Depth data stream
Audio data stream
Infrared data stream (for low light environment)
Accelerometer data 16

17. The Near Mode Track a human upper body as close as 40 cm
Only Kinect for Windows supports the near mode 17

18. Tracking Human Skeleton and Joint Movement
Kinect SDK does the magic to extract the skeleton and joint positions from the data streams received
Can track up to two users at the same time with full joint positions
Each skeleton consists of 20 joints 18

19. Building First Kinect App
Modified KinectInfoBox app
Steps
Create a new project: C# WPF application
Adding Kinect reference
Draw the GUI
Modify MainWindow.xaml
Adding code 19

20. Creating a New Project 20

21. Adding Kinect Libraries21

22. Create User Interface 22

23. Modify MainWindow.xaml23

24. Adding Code (MainWindow.xaml.cs)
Setting up name space
using Microsoft.Kinect;
Setting up state variable for the Kinect sensor
An instance of Microsoft.Kinect.KinectSensor class
It represents the complete runtime pipeline for the Kinect sensor during life cycle of the app
public partial class MainWindow : Window {
KinectSensor sensor;
// remaining code goes here } 24

25. Adding Code
private void WindowLoaded(object sender, RoutedEventArgs e) {
if (KinectSensor.KinectSensors.Count > 0) {
this.sensor = KinectSensor.KinectSensors[0];
if (this.sensor != null && !this.sensor.IsRunning) {
this.sensor.Start();
displayInfo(); }
else {
MessageBox.Show("No device is connected with system!");
this.Close();
private void WindowClosing(object sender, System.ComponentModel.CancelEventArgs e) {
if (this.sensor != null && this.sensor.IsRunning) {
this.sensor.Stop(); 25

26. Adding Code
private void displayInfo() {
this.textBlock1.Text = this.sensor.DeviceConnectionId;
this.textBlock2.Text = this.sensor.Status.ToString();
this.textBlock3.Text = this.sensor.ElevationAngle.ToString(); }
Double click “Increase” button, the template for the button1_Click() method will be automatically created
Double click “Decrease” button, the template for the button2_Click() method will be automatically created
private void button1_Click(object sender, RoutedEventArgs e) {
this.sensor.ElevationAngle = this.sensor.ElevationAngle + 1;
displayInfo(); }
private void button2_Click(object sender, RoutedEventArgs e) {
this.sensor.ElevationAngle = this.sensor.ElevationAngle - 1;
displayInfo(); } 26

27. More In-depth Stuff Starting Kinect sensor: sensor.Start()
In the Start() method
First check the status of Kinect, if no sensor connected, runtime will throw an InvalidOperationException object with the KinectNotReady message
Init the sensor if connected
Init options
None: default option
UseDepthAndPlayerIndex
UseColor
UseSkeletonTracking
UseDepth
UseAudio 27

28. More In-depth Stuff Open data streams Stopping the Kinect sensor
ColorImageStream: this.sensor.ColorStream.Enable();
DepthImageStream: this.sensor.ColorStream.Enable();
SkeletonStream: this.sensor.SkeletonStream.Enable();
Stopping the Kinect sensor
this.sensor.Stop();
Stops the color, depth, and skeleton data stream
Stops audio source, if open
Kills all threads spawned
Shuts down Kinect sensor and sets init option to None 28