1. 1 EEC-492/592 Kinect Application Development Lecture 2 Wenbing Zhao wenbing@ieee.org

2. 2 2 Outline Reminder  No class next Monday: Martin Luther King Day Components of Kinect sensor Kinect for Windows SDK Building first Kinect application

3. 3 3 Components of Kinect for Windows Color camera Infrared (IR) emitter IR depth sensor Tilt motor Microphone array LED

4. 4 4 The Color Camera Red, blue, green color images (RGB) Frame rate:  30 FPS at 640x480  12 FPS at 1280x960 Viewing range:  43 degrees vertical by 57 degrees horizontal

5. 5 5 IR Emitter & IR Depth Sensor IR emitter emits IR light in a pseudo-random dot pattern Dotted light reflects off different objects IR depth sensor reads reflected light => depth info Depth stream resolution  640x480  320x240  80x60

6. 6 6 Depth Data Processing

7. 7 7 Tilt Motor Used to change the camera and sensor’s angles to get the correct position of the human skeleton Can tilt vertically (upwards/downwards) up to 27 degrees

8. 8 8 Microphone Array 4 different microphones: locate the direction of the audio wave, not just capture the sound Enhanced noise suppression, echo cancellation, beam-forming

9. 9 9 LED Placed between camera and IR emitter Used for indicating the status of the Kinect Green indicates Kinect drivers loaded properly

10. 10 Kinect for Xbox vs. Kinect for Windows Kinect for Windows  Near mode: as close as 40cm away from Kinect  Can be used for commercial applications  Can be used in a Win7 virtual machine: HyperV, VMWare, Parallels For both  Track motion up to 12 feet (4 meters) away  Identical resolution

11. 11 Kinect for Windows SDK Supported operating systems  Windows 7, Windows 8, Windows Embedded 7 Hardware requirements  Dual core 2.66 GHz or faster CPU  Dedicated USB 2.0 bus  2GB or higher RAM Software requirement  Microsoft Visual Studio 2010 or higher  Kinect for Windows SDK: current version 1.8

12. 12 Kinect for Windows SDK

13. 13 Kinect for Windows SDK

14. 14 How Applications Interact with Kinect

15. 15 Classification of Kinect SDK APIs

16. 16 Kinect Driver The Kinect driver controls the camera, depth sensor, audio microphone array, and motion Data passes between the sensor and the app in the form of data streams  Color data stream  Depth data stream  Audio data stream  Infrared data stream (for low light environment)  Accelerameter data

17. 17 The Near Mode Track a human upper body as close as 40 cm Only Kinect for Windows supports the near mode

18. 18 Tracking Human Skeleton and Joint Movement Kinect SDK does the magic to extract the skeleton and joint positions from the data streams received Can track up to two users at the same time with full joint positions Each skeleton consists of 20 joints

19. 19 Building First Kinect App Modified KinectInfoBox app Steps  Create a new project: C# WPF application  Adding Kinect reference  Draw the GUI  Modify MainWindow.xaml  Adding code

20. 20 Creating a New Project

21. 21 Adding Kinect Libraries

22. 22 Create User Interface

23. 23 Modify MainWindow.xaml

24. 24 Adding Code (MainWindow.xaml.cs) Setting up name space  using Microsoft.Kinect; Setting up state variable for the Kinect sensor  An instance of Microsoft.Kinect.KinectSensor class  It represents the complete runtime pipeline for the Kinect sensor during life cycle of the app public partial class MainWindow : Window { KinectSensor sensor; // remaining code goes here }

25. 25 Adding Code private void WindowLoaded(object sender, RoutedEventArgs e) { if (KinectSensor.KinectSensors.Count > 0) { this.sensor = KinectSensor.KinectSensors[0]; if (this.sensor != null && !this.sensor.IsRunning) { this.sensor.Start(); displayInfo(); } else { MessageBox.Show("No device is connected with system!"); this.Close(); } private void WindowClosing(object sender, System.ComponentModel.CancelEventArgs e) { if (this.sensor != null && this.sensor.IsRunning) { this.sensor.Stop(); }

26. 26 Adding Code Double click “Increase” button, the template for the button1_Click() method will be automatically created Double click “Decrease” button, the template for the button2_Click() method will be automatically created private void displayInfo() { this.textBlock1.Text = this.sensor.DeviceConnectionId; this.textBlock2.Text = this.sensor.Status.ToString(); this.textBlock3.Text = this.sensor.ElevationAngle.ToString(); } private void button1_Click(object sender, RoutedEventArgs e) { this.sensor.ElevationAngle = this.sensor.ElevationAngle + 1; displayInfo(); } private void button2_Click(object sender, RoutedEventArgs e) { this.sensor.ElevationAngle = this.sensor.ElevationAngle - 1; displayInfo(); }

27. 27 More In-depth Stuff Starting Kinect sensor: sensor.Start() In the Start() method  First check the status of Kinect, if no sensor connected, runtime will throw an InvalidOperationException object with the KinectNotReady message  Init the sensor if connected Init options  None: default option  UseDepthAndPlayerIndex  UseColor  UseSkeletonTracking  UseDepth  UseAudio

28. 28 More In-depth Stuff Open data streams  ColorImageStream: this.sensor.ColorStream.Enable();  DepthImageStream: this.sensor.ColorStream.Enable();  SkeletonStream: this.sensor.SkeletonStream.Enable(); Stopping the Kinect sensor  this.sensor.Stop();  Stops the color, depth, and skeleton data stream  Stops audio source, if open  Kills all threads spawned  Shuts down Kinect sensor and sets init option to None