Kinect for Windows - Designing Software for Gesture & Voice Controlled User Interfaces Tim Huckaby InterKnowlogy Chairman/Founder Actus Interactive Software Founder & CEO Microsoft® Regional Director – Southern California Microsoft® MVP – Kinect for Windows Blog: team.interknowlogy.com/blogs/timhuckaby FaceBook: LinkedIn:

A little bit about me and why I am here…

Breakout Session InterKnowlogy ( Tim Huckaby, Founder/Chairman ◦13+ year old services firm – Custom Application Development ◦focused on building NUI solutions ◦Web, mobile, and desktop ◦Formal R&D Program: RECESS ◦Windows 7/8, PixelSense, WP7/8, Azure, Kinect, XAML stack - Customers: NASA to Nike; Intel to Microsoft…. About Tim, InterKnowlogy & Actus… Actus Interactive Software ( Tim Huckaby, Founder/CEO ◦Software Product Company ◦Interactive Kiosk and Active Digital Signage ◦Multi-Touch, Gesture, and Voice Driven Solutions

Relevant Use Cases (And Use Cases that do not Make Sense) Objectives And Agenda Runway to get started Technology Roadmap Where We have Been and Where we are Going Demos, Demos, Demos…. Gesture & Voice Driven User Experiences

NUI: The Natural User Interface Multi-Touch Capable Common Gestures of Touch Physical Object Interaction Virtual Object Interaction Gesture Capable Motion Based Interface Tracking of full-body movement, facial expression, and voice with precision Voice Capable Multi-array Microphone Uniquely Distinguish Audio & Voices Nueral Capable Conscious and non-conscience interfaces Non Invasive

World's Fastest-Selling Consumer Electronics Device … a Future Beyond Gaming Combination of RGB camera, depth sensor and multi-array microphone –RGB camera delivers three basic color components –Depth sensors “sees” in 3-D –Microphone locates voices by sound and excludes ambient noise Consumer Priced: $ Tracks full-body movement, facial expression, and voice with precision Kinect sensor bar scans and monitors 1 million points on the body Kinect for Windows SDK –Skeletal Tracking –Face, Gesture Recognition –Audio Echo cancellation –Audio Beam Forming –Speech Recognition –Machine Learning

Breakout Session Download the Free SDK and Developer Toolkit – us/kinectforwindows/develop/developer-downloads.aspxhttp:// us/kinectforwindows/develop/developer-downloads.aspx CodePlex Source Code Samples: – Developer Forums: – US/category/kinectsdkhttp://social.msdn.microsoft.com/Forums/en- US/category/kinectsdk Developer Center: – us/kinectforwindowsdev/default.aspx. us/kinectforwindowsdev/default.aspx How to Get Started

Breakout Session Use Case: Data Visualization & 3D Modeling Interactive data visualization 3D modeling and design Hands-off presentations Kinect Fusion

Breakout Session InterKnowlogy – Picture Viewer Tim Huckaby Founder / Chairman, InterKnowlogy Founder / CEO, Actus Software Microsoft RD & MVP

Breakout Session Kinect – 3D Anatomical Manipulation Tim Huckaby Founder / Chairman, InterKnowlogy Founder / CEO, Actus Software Microsoft RD & MVP

Breakout Session Skeletal Tracking Near Mode Depth Mode 3 Modes of Operation

Breakout Session K4W Skeletal Tracking

Breakout Session Use Cases: Exercise, Physical Therapy, Education, Training Exercise and physical therapy Senior wellness – exercise and monitoring Interactive science experiments Language/speech learning Home Healthcare / Tele-Medicine

Breakout Session The Kinect for Windows - InterKnowlogy - Physical Therapy Tim Huckaby Founder / Chairman, InterKnowlogy Founder / CEO, Actus Software Microsoft RD & MVP

Breakout Session The Good News: The Kinect SDK for Windows – free The Device is low cost The Not So Good News: Windows Only Device; You are not legally allowed to build Windows apps for your Xbox Kinect; you cannot build apps for 18 million Xbox Kinect owners with it. Not available World-wide Kinect for Windows

How to: United States - $ SKU: L6M Amazon Visit the Store Visit the Store GameStop Visit the Store Visit the Store Microsoft Store Visit the Store Visit the Store Newegg Visit the Store Visit the Store

Breakout Session Raw sensor streams Access to raw data streams from the depth sensor, color camera sensor, and four-element microphone array enables developers to build upon the low- level streams that are generated by the Kinect sensor. Skeletal tracking The capability to track the skeleton image of one or two people moving within the Kinect field of view make it easy to create gesture-driven applications. Advanced audio capabilities Audio processing capabilities include sophisticated acoustic noise suppression and echo cancellation, beam formation to identify the current sound source, and integration with the Windows speech recognition API. Sample code and documentation The SDK includes more than 100 pages of technical documentation. In addition to built-in help files, the documentation includes detailed walkthroughs for most samples provided with the SDK. Easy and familiar The SDK installs quickly, requires no complex configuration, and the complete installer size is less than 100 MB. Developers can get up and running in just a few minutes with a standard standalone Kinect sensor unit (widely available at retail outlets). Windows 7, Windows Embedded Standard, Windows 8 Desktop Apps C++, C#, or Visual Basic in Microsoft Visual Studio Free! Free SDK download, free run time license! What’s in the Kinect for Windows SDK?

Breakout Session Launch Kinect for XBox K4W Team Formed K4W Non- commercial SDK beta K4W commercial SDK TAP Initiated New K4W Hardware announced K4W commercial Release Timeline Nov, 2010 April, 2011 June, 2011 Sept, 2011 Nov, 2011 Feb 1, 2012 K4W v1.5K4W v1.6K4W v1.7 K4W Dev Center & New K4W 2.0 Hardware announced K4W v1.8 May 21, 2012 Oct, 2012 March, 2013 May, 2013 Sept, 2013

Breakout Session New background removal. An API removes the background behind the active user so that it can be replaced with an artificial background. This green-screening effect is especially useful for advertising, augmented reality gaming, training and simulation, and other immersive experiences that place the user in a different virtual environment. Realistic color capture with Kinect Fusion. A new Kinect Fusion API scans the color of the scene along with the depth information so that it can capture the color of the object along with its three-dimensional (3D) model. The API also produces a texture map for the mesh created from the scan. This feature provides a full fidelity 3D model of a scan, including color, which can be used for full color 3D printing or to create accurate 3D assets for games, CAD, and other applications. Improved tracking robustness with Kinect Fusion. This algorithm makes it easier to scan a scene. With this update, Kinect Fusion is better able to maintain its lock on the scene as the camera position moves, yielding a more reliable and consistent scanning. HTML interaction sample. This sample demonstrates implementing Kinect-enabled buttons, simple user engagement, and the use of a background removal stream in HTML5. It allows developers to use HTML5 and JavaScript to implement Kinect-enabled user interfaces, which was not possible previously—making it easier for developers to work in whatever programming languages they prefer and integrate Kinect for Windows into their existing solutions. Multiple-sensor Kinect Fusion sample. This sample shows developers how to use two sensors simultaneously to scan a person or object from both sides—making it possible to construct a 3D model without having to move the sensor or the object! Adaptive UI sample. This sample demonstrates how to build an application that adapts itself depending on the distance between the user and the screen—from gesturing at a distance to touching a touchscreen. The algorithm in this sample uses the physical dimensions and positions of the screen and sensor to determine the best ergonomic position on the screen for touch controls as well as ways the UI can adapt as the user approaches the screen or moves further away from it. K4W SDK v 1.8

Breakout Session Kinect for Windows SDK v1.8 Tim Huckaby Founder / Chairman, InterKnowlogy Founder / CEO, Actus Software Microsoft RD & MVP

Breakout Session Use Case: Interactive Media Interact with customers in-store Virtual fitting room – in-store or shop-at-home Vending machines Interactive window exhibits Augmented Reality web chat

Breakout Session Retail: Nike / Converse Tim Huckaby Founder / Chairman, InterKnowlogy Founder / CEO, Actus Software Microsoft RD & MVP

Breakout Session Perfect for Kinect / Not designed for Kinect Windows 8 / Metro / Modern UI

Breakout Session Actus Interactive Kiosk Tim Huckaby Founder / Chairman, InterKnowlogy Founder / CEO, Actus Software Microsoft RD & MVP

Breakout Session Use Case: Hands-Free Control Doctors in surgery operating medical imaging devices through gesture and voice Airplane mechanics/engine ering workers viewing instructions Cooking and other household scenarios

Breakout Session InterKnowlogy: Touch-less Operating Room Tim Huckaby Founder / Chairman, InterKnowlogy Founder / CEO, Actus Software Microsoft RD & MVP

Breakout Session Leap Motion –$70 – SoftKinetic – –$249 –Distance and Near Mode –Multiple Products –In the space for over a decade Panasonic –Works outside in direct sunlight –$2,600 Kinect for Windows Competitors

Breakout Session Carl Franklin’s Gesture Recorder GesturePak is a speech-driven gesture recording WPF app and a matcher class for.NET developers that uses the Kinect For Windows device and SDK.

Breakout Session GesturePak: Carl Franklin’s Pose/Gesture Recorder Tim Huckaby Founder / Chairman, InterKnowlogy Founder / CEO, Actus Software Microsoft RD & MVP

Breakout Session Higher fidelity HD 1080p color camera sampling at 60 FPS Noise-isolating multi-microphone array that filters ambient sounds to recognize natural speaking voices even in crowded rooms. Microsoft’s proprietary Time-of-Flight technology, which measures the time it takes individual photons to rebound off an object or person to create unprecedented accuracy and precision recognizes precise motions and details, such as slight wrist rotation, body position, and even the wrinkles in your clothes. Precision: Capable of determining your heart rate by recognizing your pulse in your face New Features Expanded Field of View Tracks 6 Skeletons at once New active infrared (IR) Works in nearly any lighting condition and, in essence, give businesses access to a new fourth sensor: audio, depth, color…and now active IR. This will offer developers better built-in recognition capabilities in different real-world settings—independent of the lighting conditions—including the sensor’s ability to recognize facial features, hand position, and more. Shipping “next year” (2014) What is Kinect For Windows 2.0?

Summary Kinect for Windows is not great in the majority of Use Cases; where it is good: Hands-Free Control Exercise, Physical Therapy, Education, Training Data Visualization, 3D Modeling Interactive Media, Retail, Digital Signage The hardware, tools, plumbing and platform for Kinect for Windows Applications are powerful and available now. …and will just continue to get better, cheaper, powerful and more available The concepts of Gesture and Voice Recognition change the way we design and build software in many Use Cases. …if the goal is better user experiences

More info on InterKnowlogy: More info on Actus Software: ◦ About Tim Huckaby… ◦Founder/Chairman, InterKnowlogy ◦Founder/Chairman, Actus Software ◦Microsoft® Regional Director – Southern California ◦Microsoft®.NET Partner Advisory Council Founder / Member ◦Microsoft® MVP – Kinect for Windows ◦INETA Speaker – International.NET Users Group Association ◦Board Member: Association for Competitive Technology ◦Board Member: Microsoft Healthcare Partner Advisory Board ◦Author / Speaker Contact me: Phone: Office: Mobile: : Blog: FaceBook: LinkedIn: Tim Huckaby