Kinect Data Sources and Programming Model Jesse Kaplan | Program Manager, Kinect Team Rob Relyea | Program Manager, Kinect Team

Rob Relyea | @rrelyea Principal Program Manager, Microsoft Member of Kinect for Windows and Xbox Teams Currently focused on Hand Pointer Gestures Deep on WPF/XAML Formerly worked as Program Manager and Architect on WPF and XAML teams

Jesse Kaplan Sr. Program Manager, Microsoft Jesse Kaplan is a Program Manager on the Kinect Team. He previously worked on .Net and the Common Language Runtime.

Kinect for Windows v2 Available Now

What’s New in K4W v2 New sensor shipping now Build 2014 11/12/2018 What’s New in K4W v2 New sensor shipping now http://aka.ms/k4wv2purchase New v2 SDK (public preview now) http://aka.ms/k4wv2previewsdk Single set of cross-platform Kinect APIs (Desktop, Store, Xbox) Create & publish Kinect apps to the Windows Store Still have robust support for WPF & Windows desktop applications Kinect Unity plugin for both desktop & store apps © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Agenda Kinect Development Overview Kinect Data Sources Other Tools Build 2014 11/12/2018 Agenda Kinect Development Overview Desktop and Windows Store Kinect Data Sources Color, Infrared, Depth, Body, BodyIndex Other Tools Frame synchronization, coordinate mapping, processing body data © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Kinect Overview Build 2014 11/12/2018 © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Kinect Data Sources Audio Build 2014 11/12/2018 © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Kinect Data Sources

Architecture, HW Spec, Design Principles Build 2014 11/12/2018 Architecture, HW Spec, Design Principles © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

High-level architecture Build 2014 11/12/2018 High-level architecture Native Apps .Net Apps Windows Store Apps C#, VB, JS, C++/CX Native API .Net API WinRT API Kinect Runtime Briefly discuss porting effort All the same – not lower-higher level Talk to USB3, Windows 8, GPU requirement Kinect Drivers Physical Kinect Sensor © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Recommended Hardware CPU -> i7 (recommended) Build 2014 11/12/2018 Recommended Hardware CPU -> i7 (recommended) RAM -> 4GB (recommended) GPU -> DirectX 11 (required) USB 3.0 -> Intel or Renesas chipsets (required) OS -> Windows 8.0/8.1 (required) © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Design Principles Minimize per-frame allocations Build 2014 11/12/2018 Design Principles Minimize per-frame allocations Low latency – high throughput Expose low level data and high level interpretations © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Creating a new store app using Kinect Build 2014 11/12/2018 Creating a new store app using Kinect In “New Project” create a new Windows Store app Enable Microphone and Webcam capabilities Add a reference to WindowsPreview.Kinect Use the WindowsPreview.Kinect namespace in your code Everything is currently “WindowsPreview.Kinect” That, of course, will change before final RTM. © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Accessing Kinect Data Sources Build 2014 11/12/2018 Accessing Kinect Data Sources © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

The KinectSensor class Build 2014 11/12/2018 The KinectSensor class Represents a single physical sensor Always valid C# JavaScript this.sensor = KinectSensor.GetDefault(); this.sensor.Open(); // Make the world a better place with Kinect this.sensor.Close(); sensor = kinect.KinectSensor.getDefault(); sensor.open(); // Make the world a better place with Kinect sensor.close(); Sensor Source Reader Frame Frame Ref © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Sources Expose metadata about the source and give access to readers Build 2014 11/12/2018 Sources Expose metadata about the source and give access to readers Sensors expose one source per data type Sensor Source Reader Frame Frame Ref © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Readers Give access to frames Build 2014 11/12/2018 Readers Give access to frames Events Polling Multiple readers may be created on a single source Readers can be paused Sensor Source Reader Frame Frame Ref © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Build 2014 11/12/2018 Readers InfraredFrameReader reader = sensor.InfraredFrameSource.OpenReader(); reader.FrameArrived += InfraredReaderFrameArrived; ... // see in the dark C# reader = sensor.infraredFrameSource.openReader(); reader.onframearrived = function (args) { ... }; ... // see in the dark JS © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Frame references Sent in frame event args Build 2014 11/12/2018 Frame references Sent in frame event args AcquireFrame gives access to the actual frame RelativeTime allow you to temporally correlate frames Sensor Source Reader Frame Frame Ref © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Build 2014 11/12/2018 Frame references void irReader_FrameArrived(InfraredFrameReader sender, InfraredFrameArrivedEventArgs args){ using (InfraredFrame frame = args.FrameReference.AcquireFrame()) { if (frame != null) // Get what you need from the frame } C# If the frame is null, that frame has expired. Your app will quickly catch up to a valid frame. var frame = args.frameReference.acquireFrame(); if(frame != null) { // Get what you need from the frame frame.close(); } JS © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Frames Gives access to the frame data Contains metadata for the frame Make a local copy or access the underlying buffer directly Contains metadata for the frame e.g. Color: format, width, height, etc. Important: Minimize how long you hold onto the frame Not Disposing frames will cause you to not receive more frames Sensor Source Reader Frame Frame Ref

Display an infrared frame in a Windows Store App

Build 2014 11/12/2018 Data Source Details Physical image sensors, ColorFrameSource, InfraredFrameSource, DepthFrameSource, BodyIndexFrameSource, BodyFrameSource, AudioFrameSource © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Two Physical “Image” Sensors Build 2014 11/12/2018 Two Physical “Image” Sensors Color Sensor 1920 x 1080 30 or 15 fps, based on lighting conditions IR Sensor 512 x 424 30 fps Different sensor means different pixel dimensions and aspect ratio. Also has wider field of view so there will be color pixels that don’t map to depth pixels. © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

ColorFrameSource 1920 x 1080 array of color pixels Build 2014 11/12/2018 ColorFrameSource 1920 x 1080 array of color pixels 30 or 15 fps, based on lighting conditions Mirrored image to match other formats © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

InfraredFrameSource Pixel Data Ambient light removed Build 2014 11/12/2018 InfraredFrameSource Pixel Data 16-bit IR intensity value Ambient light removed Not heat – different frequency People have used this for: night vision, computer vision applications that need consistent illumination like face recognition, retro reflective materials also show up very well © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

DepthFrameSource Range: 0.5 - 8 meters Pixel Data 16-bit distance in millimeters from the sensor’s focal plane

BodyIndexFrameSource Build 2014 11/12/2018 BodyIndexFrameSource Pixel Data 0 to 5: Index of the corresponding body, as tracked by the body source > 5: No tracked body at that pixel Could detect a body moved by comparing one frame to the next Can be used directly for some background removal © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

BodyFrameSource Range is 0.5-4.5 meters Build 2014 11/12/2018 BodyFrameSource Range is 0.5-4.5 meters Frame data is a collection of Body objects each with 25 joints Each joint has position in 3D space and an orientation Up to six simultaneous bodies 30fps Hand State on 2 bodies Lean © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

AudioFrameSource Data is audio samples captured over a specific interval of time Audio data is associated with an “audio beam” A steerable “cone” of focus for audio May be automatically or manually aimed

Build 2014 Coordinate mapping, MultiSourceFrameReader 11/12/2018 Other Important APIs Coordinate mapping, MultiSourceFrameReader © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Coordinate mapping Three coordinate systems Coordinate mapper provides conversions between each system Convert single or multiple points Name Applies to Dimensions Units Range Origin ColorSpacePoint Color 2 pixels 1920x1080 Top left corner DepthSpacePoint Depth, Infrared, Body index 512x424 CameraSpacePoint Body 3 meters – Infrared/depth camera CameraSpacePoint was named SkeletonPoint in the V1 API. But it applies more generally, not only to Body joint positions, but to any point that is within the depth camera’s field of view. Hence, the new name.

MultiSourceFrameReader Build 2014 11/12/2018 MultiSourceFrameReader Allows the app to get a matched set of frames from multiple sources on a single event Caveat: Delivers frames at the lowest FPS of the selected sources Replaces AllFramesReady event in V1 © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Handle body frames and coordinate mapping

Other Features Roadmap, Getting Started Build 2014 11/12/2018 Other Features Roadmap, Getting Started © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Other features: Roadmap Interactions Speech Face APIs Activities, Appearance, Expression EyeLeftClosed, EyeRightClosed, LookingAway, MouthMoved, MouthOpen, WearingGlasses, Happy expression, Neutral expression HD Face Fusion

Kinect Resources General Info & Blog -> http://kinectforwindows.com Purchase Sensor -> http://aka.ms/k4wv2purchase v2 Preview SDK -> http://aka.ms/k4wv2previewsdk Developer Forums -> http://aka.ms/k4wv2forum Twitter -> @KinectWindows, @benlower, @rrelyea Hackathon Events Dallas (July 18) -> http://aka.ms/k4wHackDallas Redmond (July 26) -> http://aka.ms/k4wHackRedmond Waterloo (August 8) -> http://aka.ms/k4wHackWaterloo

Q&A