Hand Pointer Gestures and Speech Rob Relyea | Program Manager, Kinect Team Mark Schwesinger | Program Manager, Kinect Team

Rob Relyea | @rrelyea Principal Program Manager, Microsoft Member of Kinect for Windows and Xbox Teams Currently focused on Hand Pointer Gestures Deep on WPF/XAML Formerly worked as Program Manager and Architect on WPF and XAML teams

Mark Schwesinger Sr. Program Manager, Microsoft Works on Interactions with NUI Delivered Kinect Gesture expereinces in Xbox360, Xbox One, and K4W Previous experience with a variety of Shell/App User experience teams

Module Overview Hand Pointer Gestures Speech Interaction Model Programming Model Demos Speech Enabling Speech Demo

Hand Pointer Gestures

Hand pointer gestures Engagement Targeting Press Panning/Zoom Available to desktop or store apps as an Opt-In input mechanism with Kinect for Windows 2.0

Controls Basics – XAML UserViewer | Engagement | Targeting | Press

Demo Building ControlsBasics-XAML from Scratch File New Project > Grid App Add Reference to Microsoft.Kinect.Xaml.Controls Enable Microphone + Camera capabilities for app Add KinectRegion as container for rootFrame in App.xaml.cs Run it!

Adding Hand Cursor Support Window Grid Window KinectRegion Kinect UserViewer Frame KinectRegion kinectRegion = new KinectRegion(); Grid grid = new Grid(); KinectUserViewer userViewer = new KinectUserViewer() { Height = 100, Width = 121, HorizontalAlignment = HorizontalAlignment.Center, VerticalAlignment = VerticalAlignment.Top, }; grid.Children.Add(kinectRegion); grid.Children.Add(userViewer); kinectRegion.Content = rootFrame; // Place the frame in the current Window Window.Current.Content = grid; Frame Page Page

“Engagement” for hand cursor System Engagement Manual Engagement Hands over head In certain location of room Etc… KinectRegion.SetOnePersonSystemEngagement() KinectRegion.SetTwoPersonSystemEngagement() KinectRegion.SetOnePersonManualEngagement(…) KinectRegion.SetTwoPersonManualEngagement(…)

PHiZ (Physical Interaction Zone) & KinectPointerPoint

PHIZ – Physical Interaction Zone X / Y - For Cursor Rendering KinectPointerPoint->Position->X Between 0.0 and 1.0 KinectPointerPoint->Position->Y X / Y – Allows offscreen info KinectPointerPoint->Properties->UnclampedPosition->X KinectPointerPoint->Properties->UnclampedPosition->Y Z KinectPointerPoint->Properties->HandReachExtent (raw) KinectPointerPoint->Properties->PressExtent (affected by user/UI) Correlating PointerPoint to NUI Data KinectPointerPoint->Properties->BodyTrackingId KinectPointerPoint->Properties->HandType KinectPointerPoint->Properties->BodyTimeCounter

Press Adapting to smaller visual sizes: Recommended Minimum Size: 208 x 208 (in 1080p resolution) Press attraction towards center Larger buttons will just attract away from the edge Adapting to smaller visual sizes: Make large-size hit testable Set KinectRegion.KinectPressInset (Thickness) to non-visible part of button Attraction region is smaller

Cursors

Demo Controls Basics – XAML Scroll Zoom

Interacting with Controls (in Preview release) UI Frameworks XAML (Store) WPF DirectX (Store) ControlsBasics-WPF ControlsBasics-XAML ControlsBasics-DX App (host KinectRegion, etc…) Microsoft.Kinect.Wpf.Controls Microsoft.Kinect.Xaml.Controls Controls Layer – Cursor rendering, Hit Test, Route PointerPoint via InputPointerManager Microsoft.Kinect.Toolkit.Input InputPointerManager, PressableModel, ManipulatableModel Microsoft.Kinect WindowsPreview.Kinect KinectCoreWindow.PointerMoved, KinectPointerPoint, KinectGestureRecognizer

Interacting with Content Navigating a camera through a 3d space Rotating a 3d object “Scrubbing” a video

HandPointerJoystick Demo

Speech

Speech Xbox One w/ Kinect Windows 8.0 or 8.1 w/ Kinect Engagement for Global Commands Engagement for App Commands “In Experience” Commands App- defined grammar Xbox One w/ Kinect “Xbox” “Xbox Select” App- defined grammar Windows 8.0 or 8.1 w/ Kinect “Kinect” or … N/A Available to desktop apps (ONLY) as an Opt-In input mechanism with Kinect for Windows 2.0

Speech Basics

Media Demo

Development and Runtime Requirements Development Requirements Runtime Requirements Kinect SDK Speech Platform SDK v11 (x86) Speech Platform SDK v11 (x64) Plus all Runtime Requirements Kinect Runtime Speech Platform Runtime v11 (x86) Speech Platform Runtime v11 (x64) K4W Language Models

Module Summary Hand Pointer Gestures Speech Interaction Model Programming Model Speech Enabling Speech

Kinect Resources General Info & Blog -> http://kinectforwindows.com Purchase Sensor -> http://aka.ms/k4wv2purchase v2 Preview SDK -> http://aka.ms/k4wv2previewsdk Developer Forums -> http://aka.ms/k4wv2forum Twitter -> @KinectWindows, @benlower, @rrelyea Hackathon Events Dallas (July 18) -> http://aka.ms/k4wHackDallas Redmond (July 26) -> http://aka.ms/k4wHackRedmond Waterloo (August 8) -> http://aka.ms/k4wHackWaterloo

Q&A