Fusion, Face, HD Face Matthew Simari | Program Manager, Kinect Team Ben Lower | Program Manager, Kinect Team

Ben Lower | ‏@benlower Sr. Program Manager and Developer Community Manager, Kinect for Windows Responsible for global K4W dev community Helps developers get help & resources they need to build apps & experiences Speaks at developer events about Kinect for Windows Formerly worked on Windows Phone developer marketing team

Matthew Simari | @MatthewSimari Program Manager, Microsoft Experience in a variety of computer vision fields Currently drives HD Face Tracking, Surface Reconstruction, and Camera Pose Tracking Previously worked defining / refining strong Kinect experiences on Xbox One in coordination with game developers

Module Overview Fusion Face HD Face Demo reconstructing surfaces How to make it work Demo: Fusion Explorer and Viewing 3D Models Face Core components of the API Demo: Face Sample HD Face Demo: Kinect Sports Rivals, HD Face Sample

Kinect Fusion Options, Voxels

Fusion Explorer Viewing 3D Models Fusion | Face | HD Face Fusion Explorer Viewing 3D Models

1 2 3 Fusion Option 1: Let us do the heavy lifting! Fusion | Face | HD Face Option 1: Let us do the heavy lifting! 1 Process Function Provides “six degrees of freedom” transform Can be rendered and use to begin the Fusion 2 Create Reconstruction Pass in a set of parameters defining the environment and processing type Outputs the reconstruction 3 Other considerations Can enable color to be lifted from the scan Experimental pose finders (6DoF trackers) are also included to play with

1 2 3 4 Fusion Option 2: Do it Yourself! Fusion | Face | HD Face Start Here Option 2: Do it Yourself! Alignment Performs “Iterative Closest Point” between the two point clouds Frame by frame operation within a known space Utilizes robust camera tracking 1 Integration Performs initial WorldToCamera transform Given a known world position Integrates the depth at each voxel with each integration 2 Voxel Representation Each voxel stores the data of the closest surface Defines the surfaces in a given space 3 Mesh Reconstruction Marching cubes creates the triangle based mesh Mesh provides standard outputs (i.e., numTriangles, color, etc.) 4

Fusion - - + + + + + + + + Volume + Pixel = Voxel Fusion | Face | HD Face Voxel What? Volume + Pixel = Voxel Voxels store depth information of a given pixel The are positive or negatively signed dependent on which part of a surface their on - - + + + + + + + +

Detection, Alignment, Orientation, Expressions Face Detection, Alignment, Orientation, Expressions

Face Detection Alignment Orientation Expressions Fusion | Face | HD Face Detection Outputs a bounding box around the face Can be visualized in color or IR Alignment Identifies 5 facial landmarks on the face Operation performed in color or IR Orientation Returns quaternion of the head joint with respect to the sensor Quaternion prevents gimbal lock Our alignment and detection perform the operation in IR, but are converted to color using the coordinate mapper for output Expressions Provides classifiers for happy, left / right eye open, engagement, mouth open and mouth moving

Fusion | Face | HD Face Face Samples

Face Model Builder, Face Model, Using the Mesh HD Face Face Model Builder, Face Model, Using the Mesh

HD Face 1 2 3 Fusion | Face | HD Face Face Model Builder Face Model Once of cluster of classes that implement the capture interaction Interactive API Provides collections status and evaluation of frames you’ve collected 1 Face Model Set of 94 shape units, scale, hair color, and skin color All adjustments are set against an “average face” mesh that is deformed by the shape units acquired during the face capture 2 Use the Mesh The mesh is analogous to most animation meshes for easy application to other rigs Mesh topology is the same for all faces represented in a standard way (numTriangles, numVertices, etc.) 3 UX is a non-trivial consideration in a good implementation of the Face Model Builder.

Fusion | Face | HD Face Kinect Sports Rivals

Fusion | Face | HD Face HD Face Samples

In Summary Fusion Face HD Face Provides a rich mesh reconstruction of environment and objects Surface reconstruction done using voxels Offers a great entry- point into easy to use face attributes Operations and outputs are returned in 2D space Provides a high- definition mesh of a face Reconstruction is done by deforming “shape units” Operations and outputs are returned in 3D space