1. Rob Relyea | Program Manager, Kinect Team Johan Marien | Program Manager, Kinect Team

2. Rob Relyea | @rrelyea Principal Program Manager, Microsoft –Member of Kinect for Windows and Xbox Teams –Currently focused on Hand Pointer Gestures Deep on WPF/XAML –Formerly worked as Program Manager and Architect on WPF and XAML teams

3. Johan Marien | @jmarien Sr. Program Manager, Microsoft –Johan Marien is a Program Manager on the Kinect Team working on developer tooling.

4. Kinect Studio –Recording sensor data –Playback of data Gesture Recognition –Heuristic Recognition –Machine Learning Based Recognition [w/ Gesture Builder] Module Overview

5. Kinect Studio Recording and Playback

6. DEMO Kinect Studio 2.0 Connecting | Monitoring | Configuring Views

7. Recordable Data Sources Infrared 13 MB/s Depth 13 MB/s BodyFrameBodyIndex Color 120 MB/s Audio 32 KB/s Legend Record/Play Record Only

8. DEMO Kinect Studio 2.0 Recording | Playback

9. V2.0 Now can record Audio Playback works without sensor attached API access to playback/record –Kinect Studio, only uses these public APIs! Kinect Studio – Major Improvements

10. Gesture Recognition Heuristic + Machine Learning Approaches

11. DEMO Programmatic detection of a gesture/pose

12. Gesture Builder New tool, shipping with v2 SDK Organize data using projects and solutions Give meaning to data by tagging gestures Build gestures using machine learning technology –Adaptive Boosting (AdaBoost) Trigger Determines if player is performing gesture –Random Forest Regression (RFR) Progress Determines the progress of the gesture performed by player Analyze / test the results of gesture detection Live preview of results

13. Visual Gesture Builder Rinse and repeat Your Application Record example gestures using Kinect Studio Tag gestures using Gesture Builder Build and analyze gestures using Gesture Builder Preview gestures in Gesture Builder Call gesture detector for tracked bodies

14. DEMO Machine Learning Demo

15. Heuristic Gesture is a coding problem Quick to do simple gestures/poses (hand over head) ML can also be useful to find good signals for Heuristic approach Machine Learning (ML) with G.B. Gesture is a data problem Signals which may not be easily human understandable (progress in a baseball swing) Large investment for production Danger of over-fitting, causes you to be too specific – eliminating recognition of generic cases Gesture Recognition

16. DEMO Calling gesture detector from code

17. Kinect Studio –Recording sensor data –Playback of data –APIs with Microsoft.Kinect.Tools.dll Gesture Recognition –Heuristic Recognition –Machine Learning Based Recognition [w/ Gesture Builder] Summary