Presenter: Jae Sung Park COMP 790-133 Human Pose Recognition Real-Time Human Pose Recognition in Parts from Single Depth Images Presenter: Jae Sung Park

Joint Position Proposal Results Introduction Data Acquisition Body Part Inference Joint Position Proposal Results In today’s talk, I am going to start with introduction, then go over the details of the paper, And finally I will show some results of this paper.

Poselets [Bourdev & Malik, 2009] Motivation Human body tracking has many applications gaming, telepresence, security, … RGB or intensity camera? Higher computational cost Human joint tracking can be used in many applications. Gaming is one of these applications like this. And the authors mention that using RGB or intensity camera is not appropriate For real time applications Because of higher computational cost. One example of human pose recognition using RGB camera Is an algorithm called Poselets, Poselets [Bourdev & Malik, 2009] use RGB camera Kinect game

Motivation: Kinect Kinect sensor Gets color & depth images ~30Hz frame rate Color sensor Depth sensor Color image Depth image Point cloud Position Color This is how the Kinect sensor gets color and depth images. The frame rate is about 30 frames per second, Designed for real time application.

Goal Body joint position proposal from single depth images The goal of this paper is to find the 3D joint positions from a single depth image. For training, depth images are synthesized of captured through the real Kinect, And body parts are labelled by per-pixel operations, And finally the 3D positions are proposed using mean-shift algorithm.

Joint Position Proposal Results Introduction Data Acquisition Body Part Inference Joint Position Proposal Results The first step of this algorithm is data acquisition.

Need of Data Synthesis Lack of real world data Variation of clothing, hair, body shape, camera position Generally, it is difficult to have a huge dataset from real Kinect. It would be really tedious task for human to label each body part in every images. Also, the shape of clothing, hair style, body shape and camera position Will make different looking depth images, To deal with every aspects, the size of dataset should be large.

Generation of Depth Images Camera position CMU Mocap Depth rendering So they synthesized depth images and body part labels from 3D character models and motions. CMU motion capture data or CMU mocap is used for motion generations. The motions are defined by joint angles of human. They used various character models with various clothing and hair styles, Then retargeted to the mocap data. Also they used different virtual camera locations To get depth images and body part labelled color images. Retarget Color rendering Via Texture mapping Character Models

Joint Position Proposal Results Introduction Data Acquisition Body Part Inference Joint Position Proposal Results Now I am going to talk about body part inference method.

Body Part Labeling 31 Body parts Variable depending on applications 5 for head and neck 16 for upper body 10 for lower body Variable depending on applications Sufficiently small, not too large They used 31 different body parts labelling. The labelling scheme can be different depending on applications. The number of different parts should be sufficiently small In order to localize body joints, And it also should be not too large Because the large number of body parts labelling will waste capacity of the classifier.

Depth Image Features where : feature function : depth value at pixel x in image I : two offsets in pixel space : normalization factor They defined the feature as this equation. D I of x is the depth value in the depth image at 2D pixel location x U and v are 2d offset vectors in the pixel space 1 over d of x is used for normalization.

Depth Image Features: Examples For example, The yellow cross denotes x Two arrows for theta 1 and theta 2 denote the offset vectors u and v The feature measures the depth difference between the two circled points. The left figure shows large responses because they measure depth difference between human and background depth The right figure shows small responses because of small difference between two human body pixels or between two background pixels. Large responses Small responses

Depth Image Features: Properties 1. ensures depth invariant One of the property of this feature is that It is a depth invariant feature. To make it depth invariant, normalization factor is multiplied by u and v As you can see in this figure the offset length has been changed.

Depth Image Features: Properties 2. Translation invariant measures depth difference Uses two offset vectors 3. Computationally efficient 3 image pixel reads, 5 arithmetic operations Direct implementation on GPU It is also translational invariant because First it measures depth difference, And second it uses two offset vectors. The most important thing is that this feature is computationally efficient It uses 3 image pixel fetches and 5 arithmetic operations. Which can be directly implemented on GPU.

Randomized Decision Forests Each internal node has Feature Threshold Each leaf node has distribution over body-part label They used randomized decision forests for classification. Each internal node has feature and threshold. Evaluate the feature f sub theta and If the value is less than threshold, move to the left node Otherwise move to the right node And repeat recursively until reaching to a leaf node. Leaf node has probabilistic distribution of body part label given pixel x.

Randomized Decision Forests: Learning Random subset of pixels Random subset of splitting candidates where Partition pixels into left/right subsets The learning process is as follows, First, pick a random subset of pixel locations Second, pick random candidate pairs of theta and tau, Third, partition the pixels into left and right subsets

Randomized Decision Forests: Learning Compute the giving the largest information gain where is entropy Recurse for left/right subsets Repeat 1-5 for generating several trees The next step is to find the best pair of theta and tau Which gives the largest information gain These steps are repeated until terminating condition For example reaching maximum depth. Then finally generate multiple trees.

Randomized Decision Forests: Inference Starting from each root Move to left or right node according to feature and threshold until reaching leaf node Average distribution over all trees Body part label inference is followed as I said, Evaluate function value and decide which direction to move For each tree, Then average the probabilistic distributions over all trees.

Inference Results Synthetic data Real data These are some inference results. The colors are showing the label color which has the highest probability.

Inference Results

Joint Position Proposal Results Introduction Data Acquisition Body Part Inference Joint Position Proposal Results The final step is joint position proposal.

Joint Position Proposal “Local mode-finding approach based on mean shift with weighted Gaussian kernel” Mean shift is an algorithm for locating the maxima of a density function. where : pre-learned per-part bandwidth

Joint Position Proposal Modes are on surface of body Push back in z direction by a pre-learned parameter = 0.039m = 0.065m

Introduction Data Acquisition Body Part Inference Joint Position Proposal Results

Speed Learning: Inference: 1 day on a 1000 core cluster training 3 trees, depth 20 from 1 million images Inference: Under 5ms per frame on Xbox 360 GPU

Depth of trees

Maximum Probe Offset Maximum Probe Offset = max value for offset u and v

Comparison with Other Methods Comparison with [Ganapathi et al., 2010]