1. Fusion, Face, HD Face Matthew Simari | Program Manager, Kinect Team
Ben Lower | Program Manager, Kinect Team

2. Ben Lower | ‏@benlower Sr. Program Manager and Developer
Community Manager, Kinect for Windows
Responsible for global K4W dev community
Helps developers get help & resources they need to build apps & experiences
Speaks at developer events about Kinect for Windows
Formerly worked on Windows Phone developer marketing team

3. Matthew Simari | @MatthewSimari
Program Manager, Microsoft
Experience in a variety of computer vision fields
Currently drives HD Face Tracking, Surface
Reconstruction, and Camera Pose Tracking
Previously worked defining / refining strong Kinect experiences on Xbox One in coordination with game developers

4. Module Overview Fusion Face HD Face Demo reconstructing surfaces
How to make it work
Demo: Fusion Explorer and Viewing 3D Models
Face
Core components of the API
Demo: Face Sample
HD Face
Demo: Kinect Sports Rivals, HD Face Sample

5. Kinect Fusion
Options, Voxels

6. Fusion Explorer Viewing 3D Models
Fusion | Face | HD Face
Fusion Explorer Viewing 3D Models

7. 1 2 3 Fusion Option 1: Let us do the heavy lifting!
Fusion | Face | HD Face
Option 1: Let us do the heavy lifting! 1
Process Function
Provides “six degrees of freedom” transform
Can be rendered and use to begin the Fusion 2
Create Reconstruction
Pass in a set of parameters defining the environment and processing type
Outputs the reconstruction 3
Other considerations
Can enable color to be lifted from the scan
Experimental pose finders (6DoF trackers) are also included to play with

8. 1 2 3 4 Fusion Option 2: Do it Yourself! Fusion | Face | HD Face
Start Here
Option 2: Do it Yourself!
Alignment
Performs “Iterative Closest Point” between the two point clouds
Frame by frame operation within a known space
Utilizes robust camera tracking 1
Integration
Performs initial WorldToCamera transform
Given a known world position
Integrates the depth at each voxel with each integration 2
Voxel Representation
Each voxel stores the data of the closest surface
Defines the surfaces in a given space 3
Mesh
Reconstruction
Marching cubes creates the triangle based mesh
Mesh provides standard outputs (i.e., numTriangles, color, etc.) 4

9. Fusion - - + + + + + + + + Volume + Pixel = Voxel
Fusion | Face | HD Face
Voxel What?
Volume + Pixel = Voxel
Voxels store depth information of a given pixel
The are positive or negatively signed dependent on which part of a surface their on - - + + + + + + + +

10. Detection, Alignment, Orientation, Expressions
Face
Detection, Alignment, Orientation, Expressions

11. Face Detection Alignment Orientation Expressions
Fusion | Face | HD Face
Detection
Outputs a bounding box around the face
Can be visualized in color or IR
Alignment
Identifies 5 facial landmarks on the face
Operation performed in color or IR
Orientation
Returns quaternion of the head joint with respect to the sensor
Quaternion prevents gimbal lock
Our alignment and detection perform the operation in IR, but are converted to color using the coordinate mapper for output
Expressions
Provides classifiers for happy, left / right eye open, engagement, mouth open and mouth moving

12. Fusion | Face | HD Face
Face Samples

13. Face Model Builder, Face Model, Using the Mesh
HD Face
Face Model Builder, Face Model, Using the Mesh

14. HD Face 1 2 3 Fusion | Face | HD Face Face Model Builder Face Model
Once of cluster of classes that implement the capture interaction
Interactive API
Provides collections status and evaluation of frames you’ve collected 1
Face Model
Set of 94 shape units, scale, hair color, and skin color
All adjustments are set against an “average face” mesh that is deformed by the shape units acquired during the face capture 2
Use the Mesh
The mesh is analogous to most animation meshes for easy application to other rigs
Mesh topology is the same for all faces represented in a standard way (numTriangles, numVertices, etc.) 3
UX is a non-trivial consideration in a good implementation of the Face Model Builder.

15. Fusion | Face | HD Face
Kinect Sports Rivals

16. Fusion | Face | HD Face
HD Face Samples

17. In Summary Fusion Face HD Face
Provides a rich mesh reconstruction of environment and objects
Surface reconstruction done using voxels
Offers a great entry- point into easy to use face attributes
Operations and outputs are returned in 2D space
Provides a high- definition mesh of a face
Reconstruction is done by deforming “shape units”
Operations and outputs are returned in 3D space