1. Kinect Data Sources and Programming Model
Jesse Kaplan | Program Manager, Kinect Team
Rob Relyea | Program Manager, Kinect Team

2. Rob Relyea | @rrelyea Principal Program Manager, Microsoft
Member of Kinect for Windows and Xbox Teams
Currently focused on Hand Pointer Gestures
Deep on WPF/XAML
Formerly worked as Program Manager and Architect on WPF and XAML teams

3. Jesse Kaplan Sr. Program Manager, Microsoft
Jesse Kaplan is a Program Manager on the Kinect Team. He previously worked on .Net and the Common Language Runtime.

4. Kinect for Windows v2 Available Now

5. What’s New in K4W v2 New sensor shipping now
Build 2014
11/12/2018
What’s New in K4W v2
New sensor shipping now
New v2 SDK (public preview now)
Single set of cross-platform Kinect APIs (Desktop, Store, Xbox)
Create & publish Kinect apps to the Windows Store
Still have robust support for WPF & Windows desktop applications
Kinect Unity plugin for both desktop & store apps
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

6. Agenda Kinect Development Overview Kinect Data Sources Other Tools
Build 2014
11/12/2018
Agenda
Kinect Development Overview
Desktop and Windows Store
Kinect Data Sources
Color, Infrared, Depth, Body, BodyIndex
Other Tools
Frame synchronization, coordinate mapping, processing body data
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

7. Kinect Overview Build 2014 11/12/2018
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

8. Kinect Data Sources Audio Build 2014 11/12/2018
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

9. Kinect Data Sources

10. Architecture, HW Spec, Design Principles
Build 2014
11/12/2018
Architecture, HW Spec, Design Principles
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

11. High-level architecture
Build 2014
11/12/2018
High-level architecture
Native Apps
.Net
Apps
Windows Store Apps
C#, VB, JS, C++/CX
Native API
.Net
API
WinRT
API
Kinect Runtime
Briefly discuss porting effort
All the same – not lower-higher level
Talk to USB3, Windows 8, GPU requirement
Kinect Drivers
Physical Kinect Sensor
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

12. Recommended Hardware CPU -> i7 (recommended)
Build 2014
11/12/2018
Recommended Hardware
CPU -> i7 (recommended)
RAM -> 4GB (recommended)
GPU -> DirectX 11 (required)
USB 3.0 -> Intel or Renesas chipsets (required)
OS -> Windows 8.0/8.1 (required)
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

13. Design Principles Minimize per-frame allocations
Build 2014
11/12/2018
Design Principles
Minimize per-frame allocations
Low latency – high throughput
Expose low level data and high level interpretations
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

14. Creating a new store app using Kinect
Build 2014
11/12/2018
Creating a new store app using Kinect
In “New Project” create a new Windows Store app
Enable Microphone and Webcam capabilities
Add a reference to WindowsPreview.Kinect
Use the WindowsPreview.Kinect namespace in your code
Everything is currently “WindowsPreview.Kinect” That, of course, will change before final RTM.
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

15. Accessing Kinect Data Sources
Build 2014
11/12/2018
Accessing Kinect Data Sources
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

16. The KinectSensor class
Build 2014
11/12/2018
The KinectSensor class
Represents a single physical sensor
Always valid C#
JavaScript
this.sensor = KinectSensor.GetDefault(); this.sensor.Open(); // Make the world a better place with Kinect this.sensor.Close();
sensor = kinect.KinectSensor.getDefault(); sensor.open(); // Make the world a better place with Kinect sensor.close();
Sensor
Source
Reader
Frame
Frame Ref
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

17. Sources Expose metadata about the source and give access to readers
Build 2014
11/12/2018
Sources
Expose metadata about the source and give access to readers
Sensors expose one source per data type
Sensor
Source
Reader
Frame
Frame Ref
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

18. Readers Give access to frames
Build 2014
11/12/2018
Readers
Give access to frames
Events
Polling
Multiple readers may be created on a single source
Readers can be paused
Sensor
Source
Reader
Frame
Frame Ref
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

19. Build 2014
11/12/2018
Readers
InfraredFrameReader reader = sensor.InfraredFrameSource.OpenReader();
reader.FrameArrived += InfraredReaderFrameArrived;
...
// see in the dark C#
reader = sensor.infraredFrameSource.openReader();
reader.onframearrived = function (args) { ... };
...
// see in the dark JS
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

20. Frame references Sent in frame event args
Build 2014
11/12/2018
Frame references
Sent in frame event args
AcquireFrame gives access to the actual frame
RelativeTime allow you to temporally correlate frames
Sensor
Source
Reader
Frame
Frame Ref
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

21. Build 2014
11/12/2018
Frame references
void irReader_FrameArrived(InfraredFrameReader sender,
InfraredFrameArrivedEventArgs args){
using (InfraredFrame frame = args.FrameReference.AcquireFrame()) {
if (frame != null)
// Get what you need from the frame } C#
If the frame is null, that frame has expired. Your app will quickly catch up to a valid frame.
var frame = args.frameReference.acquireFrame();
if(frame != null) {
// Get what you need from the frame
frame.close(); } JS
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

22. Frames Gives access to the frame data Contains metadata for the frame
Make a local copy or access the underlying buffer directly
Contains metadata for the frame
e.g. Color: format, width, height, etc.
Important: Minimize how long you hold onto the frame
Not Disposing frames will cause you to not receive more frames
Sensor
Source
Reader
Frame
Frame Ref

23. Display an infrared frame in a Windows Store App

24. Build 2014
11/12/2018
Data Source Details
Physical image sensors, ColorFrameSource, InfraredFrameSource, DepthFrameSource, BodyIndexFrameSource, BodyFrameSource, AudioFrameSource
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

25. Two Physical “Image” Sensors
Build 2014
11/12/2018
Two Physical “Image” Sensors
Color Sensor
1920 x 1080
30 or 15 fps, based on lighting conditions
IR Sensor
512 x 424
30 fps
Different sensor means different pixel dimensions and aspect ratio. Also has wider field of view so there will be color pixels that don’t map to depth pixels.
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

26. ColorFrameSource 1920 x 1080 array of color pixels
Build 2014
11/12/2018
ColorFrameSource
1920 x 1080 array of color pixels
30 or 15 fps, based on lighting conditions
Mirrored image to match other formats
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

27. InfraredFrameSource Pixel Data Ambient light removed
Build 2014
11/12/2018
InfraredFrameSource
Pixel Data
16-bit IR intensity value
Ambient light removed
Not heat – different frequency
People have used this for: night vision, computer vision applications that need consistent illumination like face recognition, retro reflective materials also show up very well
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

28. DepthFrameSource Range: 0.5 - 8 meters Pixel Data
16-bit distance in millimeters from the sensor’s focal plane

29. BodyIndexFrameSource
Build 2014
11/12/2018
BodyIndexFrameSource
Pixel Data
0 to 5: Index of the corresponding body, as tracked by the body source
> 5: No tracked body at that pixel
Could detect a body moved by comparing one frame to the next
Can be used directly for some background removal
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

30. BodyFrameSource Range is 0.5-4.5 meters
Build 2014
11/12/2018
BodyFrameSource
Range is meters
Frame data is a collection of Body objects each with 25 joints
Each joint has position in 3D space and an orientation
Up to six simultaneous bodies
30fps
Hand State on 2 bodies
Lean
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

31. AudioFrameSource
Data is audio samples captured over a specific interval of time
Audio data is associated with an “audio beam”
A steerable “cone” of focus for audio
May be automatically or manually aimed

32. Build 2014 Coordinate mapping, MultiSourceFrameReader
11/12/2018
Other Important APIs
Coordinate mapping, MultiSourceFrameReader
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

33. Coordinate mapping Three coordinate systems
Coordinate mapper provides conversions between each system
Convert single or multiple points
Name
Applies to
Dimensions
Units
Range
Origin
ColorSpacePoint
Color 2
pixels
1920x1080
Top left corner
DepthSpacePoint
Depth, Infrared, Body index
512x424
CameraSpacePoint
Body 3
meters –
Infrared/depth camera
CameraSpacePoint was named SkeletonPoint in the V1 API. But it applies more generally, not only to Body joint positions, but to any point that is within the depth camera’s field of view. Hence, the new name.

34. MultiSourceFrameReader
Build 2014
11/12/2018
MultiSourceFrameReader
Allows the app to get a matched set of frames from multiple sources on a single event
Caveat: Delivers frames at the lowest FPS of the selected sources
Replaces AllFramesReady event in V1
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

35. Handle body frames and coordinate mapping

36. Other Features Roadmap, Getting Started
Build 2014
11/12/2018
Other Features Roadmap, Getting Started
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

37. Other features: Roadmap
Interactions
Speech
Face APIs
Activities, Appearance, Expression
EyeLeftClosed, EyeRightClosed, LookingAway, MouthMoved, MouthOpen, WearingGlasses, Happy expression, Neutral expression
HD Face
Fusion

38. Kinect Resources General Info & Blog -> http://kinectforwindows.com
Purchase Sensor ->
v2 Preview SDK ->
Developer Forums ->
@rrelyea
Hackathon Events
Dallas (July 18) ->
Redmond (July 26) ->
Waterloo (August 8) ->

39. Q&A