Microsoft Kinect How does a machine infer body position?

Disclaimer A lot of the information presented comes from reverse engineering performed by a number of agencies. So they might not reflect true ground data, which is proprietary and thus not publicly available. 1

Microsoft Kinect  Series of motion sensing input devices for the Xbox  Launched in November 2010  Removes the need of a game controller  User interacts through gestures and spoken commands. 2

Two step Process  Compute Depth Map  Apply Machine Learning 3

Depth Map? 4

So how does it manage something so cool? 5

Projects a speckle pattern of Infrared Laser Light Depths Inferred from the distortions seen in the pattern 6

● Depth from Focus ○ What is far will be more blurry ○ Combine different focal lengths for X Axis and Y Axis ● Depth from Stereo ○ Combine image obtained from cameras in different positions ○ The speckle pattern projected from one point, observed from another (Cont) The cool combined with the Classic 7

We have the depth. What do we do next? Infer the Body Position, of course 8

Body Position Inference  Kinect starts with millions of labelled depth images  Learns a randomized decision forest, mapping depth images to body parts  This RDF takes days to be trained, but the process is made faster using parallel computing 9

Body Position Inference  Kinect uses the depth image and the RDF and computes a probability of a pixel belonging to each body part.  Uses mean shift algorithm to select the body parts and sketches a skeleton.  All of these computations are very fast. 10

Advantages  Works separately per frame as all the computations are very fast  Runs at 31 fps  Significantly more efficient and robust than body trackers  Works real-time with great results  Handles multiple players  Reported Accuracy: ~72% 11

References explained.html

Thank You Arhan Sibal Rudresh Pande