1. Hand Pointer Gestures and Speech
Rob Relyea | Program Manager, Kinect Team
Mark Schwesinger | Program Manager, Kinect Team

2. Rob Relyea | @rrelyea Principal Program Manager, Microsoft
Member of Kinect for Windows and Xbox Teams
Currently focused on Hand Pointer Gestures
Deep on WPF/XAML
Formerly worked as Program Manager and Architect on WPF and XAML teams

3. Mark Schwesinger Sr. Program Manager, Microsoft
Works on Interactions with NUI
Delivered Kinect Gesture expereinces in Xbox360,
Xbox One, and K4W
Previous experience with a variety of Shell/App User experience teams

4. Module Overview Hand Pointer Gestures Speech Interaction Model
Programming Model
Demos
Speech
Enabling Speech
Demo

5. Hand Pointer Gestures

6. Hand pointer gestures
Engagement
Targeting
Press
Panning/Zoom
Available to desktop or store apps as an Opt-In input mechanism with Kinect for Windows 2.0

7. Controls Basics – XAML UserViewer | Engagement | Targeting | Press

8. Demo Building ControlsBasics-XAML from Scratch
File New Project > Grid App
Add Reference to Microsoft.Kinect.Xaml.Controls
Enable Microphone + Camera capabilities for app
Add KinectRegion as container for rootFrame in App.xaml.cs
Run it!

9. Adding Hand Cursor Support
Window
Grid
Window
KinectRegion
Kinect UserViewer
Frame
KinectRegion kinectRegion = new KinectRegion();
Grid grid = new Grid();
KinectUserViewer userViewer = new KinectUserViewer() {
Height = 100,
Width = 121,
HorizontalAlignment = HorizontalAlignment.Center,
VerticalAlignment = VerticalAlignment.Top, };
grid.Children.Add(kinectRegion);
grid.Children.Add(userViewer);
kinectRegion.Content = rootFrame;
// Place the frame in the current Window
Window.Current.Content = grid;
Frame
Page
Page

10. “Engagement” for hand cursor
System Engagement
Manual Engagement
Hands over head
In certain location of room
Etc…
KinectRegion.SetOnePersonSystemEngagement()
KinectRegion.SetTwoPersonSystemEngagement()
KinectRegion.SetOnePersonManualEngagement(…)
KinectRegion.SetTwoPersonManualEngagement(…)

11. PHiZ (Physical Interaction Zone) & KinectPointerPoint

12. PHIZ – Physical Interaction Zone
X / Y - For Cursor Rendering
KinectPointerPoint->Position->X
Between 0.0 and 1.0
KinectPointerPoint->Position->Y
X / Y – Allows offscreen info
KinectPointerPoint->Properties->UnclampedPosition->X
KinectPointerPoint->Properties->UnclampedPosition->Y Z
KinectPointerPoint->Properties->HandReachExtent (raw)
KinectPointerPoint->Properties->PressExtent (affected by user/UI)
Correlating PointerPoint to NUI Data
KinectPointerPoint->Properties->BodyTrackingId
KinectPointerPoint->Properties->HandType
KinectPointerPoint->Properties->BodyTimeCounter

13. Press Adapting to smaller visual sizes: Recommended Minimum Size:
208 x 208 (in 1080p resolution)
Press attraction towards center
Larger buttons will just attract away from the edge
Adapting to smaller visual sizes:
Make large-size hit testable
Set KinectRegion.KinectPressInset (Thickness) to non-visible part of button
Attraction region is smaller

14. Cursors

15. Demo Controls Basics – XAML
Scroll
Zoom

16. Interacting with Controls (in Preview release)
UI Frameworks
XAML (Store)
WPF
DirectX (Store)
ControlsBasics-WPF
ControlsBasics-XAML
ControlsBasics-DX
App (host KinectRegion, etc…)
Microsoft.Kinect.Wpf.Controls
Microsoft.Kinect.Xaml.Controls
Controls Layer – Cursor rendering, Hit Test, Route PointerPoint via InputPointerManager
Microsoft.Kinect.Toolkit.Input
InputPointerManager, PressableModel, ManipulatableModel
Microsoft.Kinect
WindowsPreview.Kinect
KinectCoreWindow.PointerMoved,
KinectPointerPoint, KinectGestureRecognizer

17. Interacting with Content
Navigating a camera through a 3d space
Rotating a 3d object
“Scrubbing” a video

18. HandPointerJoystick Demo

19. Speech

20. Speech Xbox One w/ Kinect Windows 8.0 or 8.1 w/ Kinect
Engagement for Global Commands
Engagement for App Commands
“In Experience” Commands
App- defined grammar
Xbox One w/ Kinect
“Xbox”
“Xbox Select”
App- defined grammar
Windows 8.0 or 8.1
w/ Kinect
“Kinect” or …
N/A
Available to desktop apps (ONLY) as an Opt-In input mechanism with Kinect for Windows 2.0

21. Speech Basics

22. Media Demo

23. Development and Runtime Requirements
Development Requirements
Runtime Requirements
Kinect SDK
Speech Platform SDK v11 (x86)
Speech Platform SDK v11 (x64)
Plus all Runtime Requirements
Kinect Runtime
Speech Platform Runtime v11 (x86)
Speech Platform Runtime v11 (x64)
K4W Language Models

24. Module Summary Hand Pointer Gestures Speech Interaction Model
Programming Model
Speech
Enabling Speech

25. Kinect Resources General Info & Blog -> http://kinectforwindows.com
Purchase Sensor ->
v2 Preview SDK ->
Developer Forums ->
@rrelyea
Hackathon Events
Dallas (July 18) ->
Redmond (July 26) ->
Waterloo (August 8) ->

26. Q&A