Creative Coding & the New Kinect Ben Lower, Kinect Community Evangelism Carmine Sirignano, Developer Support Engineer

01 | Shifting the Microsoft Culture Legos vs. Clay

02 | Shifting the Microsoft Culture Legos versus Clay

Creativity = Exploration + Experimentation + Freedom to Fail + Fun 01 | Shifting the Microsoft Culture Creativity = Exploration + Experimentation + Freedom to Fail + Fun

Kinect Common Bridge & Unity Plug-in 01 | Shifting the Microsoft Culture Kinect Common Bridge & Unity Plug-in

Kinect Overview

Welcome to Kinect A new kind of Camera, that can see in 3D Build 2014 12/29/2017 Welcome to Kinect A new kind of Camera, that can see in 3D A traditional video camera A 4-element microphone array A powerful SDK which can: Recognize that a person is in front of the sensor, and what position their body is in A software controllable directional microphone, and speech recognition technology And lots more © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Build 2014 12/29/2017 What’s New in K4W v2 Single set of cross-platform Kinect APIs (Desktop, Store, Xbox) Create & publish Kinect apps to the Windows Store Still have robust support for WPF & Windows desktop applications Kinect Unity plugin for both desktop & store apps Kinect Common Bridge for openFrameworks and Cinder © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Sensor Features, Sensor Components Sensor Overview Sensor Features, Sensor Components

Kinect for Windows v2 Sensor Features Wider field of view (horizontally and vertically) for depth and color 3x depth fidelity 1080p, HD color camera (30fps) Lighting independent IR (30 fps) Improved microphones (zero balanced) 25 skeletal joints for six people -> Bio correct skeletal joints (particularly hips/shoulders/spine) plus more points and rotations Thumb tracking, end of hand tracking, Open/Closed Improved range -> .5M near, 4.5M far (extended depth to 8M)

Sensor Components Depth Sensor Power Light IR Emitters RGB Camera Image: iFixit, http://www.ifixit.com Power Light RGB Camera Microphone Array

Connecting the Sensor SDK Browser (Samples & Tools)

Color, Infrared, Depth, BodyIndex, Body, Audio Kinect Data Sources Color, Infrared, Depth, BodyIndex, Body, Audio

Kinect Data Sources Audio Infrared Depth Color BodyIndex Body Build 2014 12/29/2017 Kinect Data Sources Infrared Depth Color BodyIndex Body Called “Streams” in V1 Audio © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Kinect Evolution

Recommended Hardware CPU -> i7 (recommended) Build 2014 12/29/2017 Recommended Hardware CPU -> i7 (recommended) RAM -> 4GB (recommended) GPU -> DirectX 11 (required) USB 3.0 -> Intel or Renesas chipsets (required) OS -> Windows 8.0/8.1 (required) © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Build 2014 12/29/2017 Data Source Details Physical image sensors, ColorFrameSource, InfraredFrameSource, DepthFrameSource, BodyIndexFrameSource, BodyFrameSource, AudioFrameSource © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Two Physical “Image” Sensors Build 2014 12/29/2017 Two Physical “Image” Sensors Color Sensor 1920 x 1080 30 or 15 fps, based on lighting conditions IR Sensor 512 x 424 30 fps Different sensor means different pixel dimensions and aspect ratio. Also has wider field of view so there will be color pixels that don’t map to depth pixels. © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

ColorFrameSource 1920 x 1080 array of color pixels Build 2014 12/29/2017 ColorFrameSource 1920 x 1080 array of color pixels 30 or 15 fps, based on lighting conditions Mirrored image to match other formats © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

InfraredFrameSource Pixel Data Ambient light removed Build 2014 12/29/2017 InfraredFrameSource Pixel Data 16-bit IR intensity value Ambient light removed Not heat – different frequency People have used this for: night vision, computer vision applications that need consistent illumination like face recognition, retro reflective materials also show up very well © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

DepthFrameSource Range: 0.5 - 8 meters Pixel Data 16-bit distance in millimeters from the sensor’s focal plane

BodyIndexFrameSource Build 2014 12/29/2017 BodyIndexFrameSource Pixel Data 0 to 5: Index of the corresponding body, as tracked by the body source > 5: No tracked body at that pixel Could detect a body moved by comparing one frame to the next Can be used directly for some background removal © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

BodyFrameSource Range is 0.5-4.5 meters Build 2014 12/29/2017 BodyFrameSource Range is 0.5-4.5 meters Frame data is a collection of Body objects each with 25 joints Each joint has position in 3D space and an orientation Up to six simultaneous bodies 30fps Hand State on 2 bodies Lean © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Build 2014 Jumpstart videos: http://aka.ms/k4wv2jumpstart 12/29/2017 Other Features, Resources Jumpstart videos: http://aka.ms/k4wv2jumpstart © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Hand Pointer Gestures and Speech

Fusion, Face, HD Face

Recording, Playback, and Gesture Recognition

Kinect Studio Visual Gesture Builder Custom Gesture Detection Interactions Fusion

Thank You! Kinect Resources SDK & Libraries Events, Sensors, General Info K4W v2 SDK -> http://aka.ms/k4wv2previewsdk Kinect Common Bridge -> http://aka.ms/KCBv2 Cinder -> http://aka.ms/CinderKCBv2 openFrameworks -> http://aka.ms/oFKCBv2 Creative Coding Setup -> http://aka.ms/k4wccsetup Metrilus Finger Tracking -> http://aka.ms/aiolosk4wv2 Vancouver (Nov 8) -> http://aka.ms/k4whackvan Purchase Sensor -> http://aka.ms/k4wv2purchase General Info & Blog -> http://kinectforwindows.com Tutorials, Content, Support Forum 7 Hours of Deep Content -> http://aka.ms/k4wv2jumpstart VGB Deep Dive -> http://aka.ms/k4wv2vgbdeepdive Windows Store oF -> http://aka.ms/OFWinrtDemos Developer Forums -> http://aka.ms/k4wv2forum Thank You! @KinectWindows, @benlower, @rickbarraza We want your feedback and pull requests!