3d Pose Detection Used by Kinect Accurate when the pose closely matches a stored pose Inaccurate when novel poses are made Can often produce shaky movement due to pose snapping

3d Pose Tracking Calculate poses based on previous poses and current data No datasets required Has issues with local minima

Hybrid Store a collection of preprocessed poses Start session with a detected pose Track until a failure is hit Detect that frame, and continue tracking from there

Pose Tracking in Detail Given the current set of pixels, and the calculated poses of the past few frames, calculate the most probable current pose Repeat the following until error is reduced Calculate a hypothesized 3d representation (rendered image) using the 3d camera image (observed data) Calculate quite a lot about the rendered image: Edepth, Eextra, Esilhouette, Eprior Adjust the hypothesized pose accordingly If that fails significantly, “detect” the current pose and try again

Optimizing the Position Solve energy minimization problems: Edepth: Depth data, excluding any rendered foreground pixels that match background observed pixels Eextra: The above depth term, but with added calculations where rendered pixels don’t match with observed pixels Esilhouette: The difference in silhouettes (pixel is in foreground of one image but the background of the other) Eprior: Evaluation of the rendered pose based on the previous rendered poses

More Pose Detecting Grab a random subset of pixels and match them to known poses Throw out uncertain data 20k poses total

Setup User must stand in a specific pose for calibration Select random subset of pixels to determine widths of limbs

Results

Results Roughly 1 in 200 frames reinitialized by the detector Very fast movement often causes local minima On average, 1 failure every 33 seconds High performance running in parallel on the GPU

http://www.youtube.com/watch?v=ICFKEOk3SyA&feature=youtu.be

Further Work Use skinned mesh models Increase pose set Use color data in the algorithm as well