1. Advanced Topics: Skeletal Tracking and Depth Filtering
Casey Meekhof | Program Manager, Kinect Team
Ben Lower | Program Manager, Kinect Team

2. Ben Lower | ‏@benlower Sr. Program Manager and Developer
Community Manager, Kinect for Windows
Responsible for global K4W dev community
Helps developers get help & resources they need to build apps & experiences
Speaks at developer events about Kinect for Windows
Formerly worked on Windows Phone developer marketing team

3. Casey Meekhof | @kcmeeks
Principal Program Manager, Microsoft
Works on depth generation and skeletal tracking
Background in graphics and game development
Builds lots of prototypes
Had pancakes this morning

4. Application processing
11/12/2018
Gesture lifecycle
Goal
Kinect platform
Application processing
Talk focus
Correct action
Hand State
Depth
Gesture recognition
Happy user
Express-ions
Incorrect
action
Pointer ST
Angry user
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

5. Getting the most out of the platform
11/12/2018
Getting the most out of the platform
K4W features provide a great start
Depth/IR sensor plus skeletal tracking, hand tracking, face tracking, etc…
Will help you get up and running very quickly
But they are not a complete solution for every use case
With some custom filtering, your experience can be great
Use each feature for it’s strengths
Fine tune for your specific scenario
Expect to make tradeoffs
We’ll look at two case studies
System cursor
Motion detector
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

6. Case Study #1: System Cursor
Building the Cursor, Basic Algorithm, Anchor Movement Problem & Filtering, PHIZ Orientation & Sizing, Hand Points

7. Building the system cursor
11/12/2018
Building the system cursor
The first thing we started on
Not just about UI!
High-precision hand point
Body-relative positioning
Movement range in front of body (known as the PHIZ)
Example use cases:
Steering
Aiming at a target
Pushing, swiping, or gripping gestures
Interacting with a 3D environment
Techniques apply to other skeleton uses
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

8. Basic cursor algorithm
11/12/2018
Basic cursor algorithm
Calculate PHIZ center
Choose anchor point on body
Offset anchor to desired center
Determine boundaries
Orientation relative to body
Scale size based on body size
Map hand point into PHIZ
Calculate hand delta from center
Scale/offset based on boundaries
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

9. The anchor movement problem
11/12/2018
The anchor movement problem
When the anchor point moves, the PHIZ moves
Keeps the cursor relative to the body
Great if the person is moving
Core body joints can jump under occlusion
Even when standing still
Causes false “wandering” motion
So we need to filter the anchor
Tradeoff of cursor stability vs. PHIZ accuracy
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

10. Anchor point filtering
11/12/2018
Anchor point filtering
Accumulated average works well
Average the position for every frame indefinitely
anchor = lastAnchor*count/(count+1) + newPos*(1/(count+1))
++count;
Reset when current frame position is far from the accumulated average
if(XMVectorGetX(XMVector3Length(newPos- anchor)) > 0.15f) {
anchor = newPos;
count = 1;}
Trades PHIZ location accuracy for cursor stability
Leaning will cause cursor movement
Often acceptable tradeoff when precision is the top priority
Note that we tried lots of different techniques (rolling average, opposite shoulder, shoulder latch).
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

11. Calculating PHIZ orientation
11/12/2018
Calculating PHIZ orientation
Recommendation: don’t worry about body orientation
More inputs add more noise
Orient the PHIZ to always face the sensor
Align the y-axis to gravity
Tilt correct with FloorClipPlane.xyz
Falls back to accelerometer when no floor is found
Rotate scene such that “up” is (0,1,0)
Align x-axis with direction to sensor
Calculate direction to sensor
Direction from tilt-corrected PHIZ center to (0,0,0)
Cross with (0,1,0) to get x-axis
x-axis
Sensor dir
Note that it is important to always do tilt correction to normalize sensor tilt.
Also note that sensor direction does change based on user location.
Display and sensor
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

12. Determining PHIZ size Choose base size for your interaction
11/12/2018
Determining PHIZ size
Choose base size for your interaction
Smaller PHIZ increases sensitivity
Larger PHIZ suppresses noise
Still elbow minimizes fatigue
Scale up or down based on user size
Forearm length is ideal
But elbow and wrists aren’t always visible
Shoulder-to-shoulder is more reliable
But may not correlate with arm length
Lock the size as early as possible
Frame over frame deltas create cursor movement
Once you set it, leave it. Better to be a little bit off than updating all the time.
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

13. Choosing a hand point to track
11/12/2018
Choosing a hand point to track
Hand center is great for precision
One of the most precise joints
Side note: head is also quite good
Also consider the hand tip
Almost as precise as the hand
More sensitive to slight movements
Generates more movement on hand open/close
Don’t use wrists
They tend to jump under occlusion
Original Kinect advice (average wrists with hands) no longer applies
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

14. Hand point filtering Goal: 1-to-1 motion
11/12/2018
Hand point filtering
Goal: 1-to-1 motion
No movement when hand is still
Quick response when hand moves
Cursor stops when hand stops
We like the double exponential filter
Suppresses motion within a radius (recommend 5 cm)
Dampens movement outside primary motion direction
Latency is tune-able
Source available upon request
Prediction tuning required
Can cause overshooting on sudden stops
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

15. House of Cursors

16. The same concepts apply to lean tracking
11/12/2018
The same concepts apply to lean tracking
Accumulation filter on the hips
They get occluded and can shift while seated
Assume the user is facing the sensor
Minimize false motion due to extra noise
Double exponential on the head/neck
Or try your own techniques (quantization and so on)
Available in the body frame!
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

17. Case Study #2: Motion Detector
Why use Depth for Gestures, Motion Detector Inputs & Algorithm, Counting “Moved” Pixels, Zero Pixels, Jumpy Pixels, Noise Floor

18. Why use depth for gestures?
11/12/2018
Why use depth for gestures?
Go beyond platform capabilities
Track people in unsupported poses
Lying down, leaning way back, legs up
Track new things
Props, fingers
Work around gesture detection issues
Smooth out jittery inputs
Sample depth for extra smoothing
Suppress false motion
Check depth for movement as a “2nd opinion”
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

19. Case study: motion detection
11/12/2018
Case study: motion detection
Controller-based game with “escape grapple” gesture
ST didn’t work for some cases
Goal was to support very laid back poses
False positives and false negatives were too high
Wanted any motion to work
Doesn’t matter which part of the body moves
Raise controller, head-butt, kick, flail arms…
Motion detection was a perfect fit
Scenario didn’t require a skeleton at all
Not using ST improved reliability in hard poses
Dramatic reenactment. Not actual gameplay footage.
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

20. Motion detector inputs
11/12/2018
Motion detector inputs
Depth image
Distance from sensor (in mm) of every pixel
Segmentation mask
Also known as BodyIndexFrame
Tracks which pixels are users and which are background
Active body index
Which person to track (0-5)
You get to decide
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

21. Motion detector algorithm
11/12/2018
Motion detector algorithm
For every pixel in the segmentation mask
Compare depth value to the last frame’s value
If delta is greater than some threshold, mark as “moved”
Count the number of “moved” pixels
If greater than some threshold, the person moved
Easy algorithm, but there are challenges
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

22. Counting “moved” pixels
11/12/2018
Counting “moved” pixels
Counting pixels can make you resolution-dependent
People cover fewer pixels when farther away
Causes changes to gesture sensitivity at different distances
Recommendation: use area (mm2)
Resolution-independent size measurement
area = (avgDepth*inverseFocalLength)2 * pixelCount
Approximate inverseFocalLength is
Can sometimes get away with percentage
Number of “moved” pixels, divided by body mask pixels
Note that that the legs won’t always be in the mask
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

23. The zero pixel challenge
11/12/2018
The zero pixel challenge
Zero pixels can mean two things
Low confidence (bad pixel)
Light return was low (shadow, windows, leather pants / couches)
Light return didn’t make sense (edges, reflective t-shirts)
Depth was past 4.5 meters
The depth image doesn’t say which is which
Will cause false motion if you don’t filter
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

24. Filtering zero pixels Option 1: Skip them
11/12/2018
Filtering zero pixels
Option 1: Skip them
If the last frame or current frame were zero, don’t count this pixel
Works great for motion in front of body
But mutes motion on silhouette edges
Option 2: Try to figure out which is which
For each pixel, count the consecutive zero frames
If a pixel had more than 10-15, treat as maximum depth; otherwise, ignore
Brings back silhouette edges, but costs CPU and memory
We used option 1 for the motion detector
For a controller-based game, most motion starts from in front of body
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

25. Filtering jumpy pixels
11/12/2018
Filtering jumpy pixels
Some pixels jump forward and back each frame
Even while the user is holding still
Can trigger delta threshold and cause false motion
Reduce this with depth delta cancellation
Add two frames worth of deltas
Frame over frame movement cancels itself out
delta = fabs((depthN – depthN-1) + (depthN-1 – depthN-2))
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

26. Motion area filtering Some rooms have more baseline noise
11/12/2018
Motion area filtering
Some rooms have more baseline noise
Even after all the pixel-level filtering
Fixed area threshold doesn’t always work
You’ll have false positives in noisy rooms
Recommendation: adaptive threshold
Keep a running average and standard deviation of the motion area
Trigger the gesture whenever area spikes far enough above this “noise floor”
if(area > areaAvg + 2*stdDev + areaThreshold)
didPersonMove = true;
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

27. Managing the noise floor
11/12/2018
Managing the noise floor
Real motion causes noise floor to increase
Under continuous motion, detector will stop working
Recommendation: rise slow and fall fast
When the user is moving, noise floor should take a while to catch up
When the user is not moving, noise floor should drop as quickly as possible
Algorithm
For each new frame area…
If above the noise floor, only increase noise floor by 1/100th of the delta
If far below the noise floor for 10 frames in a row, reset to that new low
Otherwise, just average the new area into the running average and std dev
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

28. Motion Man

29. Motion detector use cases
11/12/2018
Motion detector use cases
An extra button for games
More control without taking hands off the gamepad
Very little motion required
Suppress false gesture detection
Ignore ST joint movement if nearby depth is quiet
Extend to do more
Detect which side the motion was on (turns 1 button into 3+)
Detect direction of motion (push and swipe gestures)
Lightweight point tracking (analog control in poses beyond ST)
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

30. Kinect Resources General Info & Blog -> http://kinectforwindows.com
Purchase Sensor ->
v2 Preview SDK ->
Developer Forums ->
@rrelyea
Hackathon Events
Dallas (July 18) ->
Redmond (July 26) ->
Waterloo (August 8) ->