Segmentation and tracking of the upper body model from range data with applications in hand gesture recognition Navin Goel Intel Corporation Department of Computer Science, University of Nevada, Reno

Overview n Introduction n Overall System n Upper Body Model n Segmentation Problem n Tracking n Color Based Segmentation n Results n Conclusion and Future Work

Introduction n Applications 3D editing system/ HCI systems, American Sign Language Recognition, Entertainment, Industrial Control, Video coding, teleconferencing n Requirements Background and illumination independent, Occlusions and self occlusions of the body components, Robust hand free initialization, Robust tracking.

Overall System Initial Segmentation Tracking Stereo (RGB+Z) video sequence Valid Track Invalid Track Color-based segmentation Hue Moments Calculation Train Reco Upper Body Model Color video sequence

Upper Body Model Ha l J C O O ij L FlFl UlUl HeT UrUr FrFr Ha r L Ha L He LTLT LULU WlWl ElEl SlSl NSrSr ErEr WrWr L Ha LFLF LULU LFLF

Head — Normal component model Upper Body Model Size Head Neck Planar component model Neck Width Torso Linear component models Elbow Wrist

Upper Body Model Linear PDF Parameters: Where, are the spherical coordinates of J c with the origin in J p The conditional probability of a joint Jc given its parent joint Jp and the anthropological measure L is given by: Where, K J c is a normalization constant, represent the minimum and maximum values of parameters

state assignments and joint for the arm and body (head &torso) regions. Stage IStage II Looking for all possible joint configuration is computationally impractical. Therefore, segmentation takes place in two stages. The Segmentation Problem Simplifying assumptions Notations Only one user is visible and his/hers torso is the largest body component, The torso plane is perpendicular to the camera and, Head is in vertical position.

Step 3 Compute Step 4 Estimate the joints: Step 1 Estimate the torso plane parameters from all data using EM. Estimate the torso and head bounding box, and the plane that includes N. Step 2 Estimate the head blob parameters from all data using EM. Step 5 Repeat steps 3-4 until convergence of The Upper Body Segmentation. Stage I

Step 1. For each possible arm parameters estimate the mean of the linear pdfs corresponding to the upper and fore arms, and the mean of the normal pdf for the hands, Step 2. For each joint configuration J A : a) compute the best state assignment of the observation vectors given the joint configuration, b) compute the observation likelihood given the joint configuration. Step 3. Find the max likelihood over all joint configuration and determine the “best” set of joints and the corresponding best state assignment. Given the fix positions of S l and S r, we sub sample the joint space to get N E =18 possible positions for each of the joints E l and E r. Given each position of the elbow joints we search for N W = 16 possible positions for each of the joints W l, W r. The Upper Body Segmentation. Stage II

Arm Tracking for each joint J p we build a set of [J c 1, J c 2, J c 3, J c 4, J c 5 ] five possible child joint positions such that each of them lies on the surface of the sphere with parent joint as the center. Z Y X Φ θ J c 1 = (r,Φ,θ) joint center from last frame J c3 = (r,Φ,θ+Δθ) J c 5 = (r,Φ+ΔΦ,θ) J c 4 = (r,Φ,θ-Δθ) Step 2 for each joint configuration we determine the best state assignment of the observations J c 2 = (r,Φ-ΔΦ,θ) Jc1Jc1 Jc2Jc2 Jc3Jc3 Jc5Jc5 Jc4Jc4 Step 3 the max log likelihood determines the best joint configuration. Step 1 estimate the mean of the linear pdfs corresponding to the upper and fore arms, and the mean of the normal pdf for the hands

Color Based Segmentation Pixels with no depth information cannot be assigned to body components by the previous segmentation algorithm. Need to estimate the depth of all pixels and perform global segmentation. Depth Segmentation

Color Based Segmentation In practice Suppose, k = “left forearm”, then l = “all the body components except left forearm”, and if Z k = “a” then Z l = “[z min … z max ] > a’’. Color Segmentation

Upper Body Segmentation and Tracking. Results

Contributions n Articulated upper body model from dense disparity maps, n Linear pdf for the fore arms and upper arms, n Hand free initialization of the system from the optimal joint configuration, n Upper body tracking, seen as a particular case of the initialization. Future work n Improvements to the background segmentation, n Learn the anthropological measures, n Integration with other HCI systems (gesture reco, face reco, speech reco, speaker identification etc.) Conclusion and Future Work