Microsoft Kinect Jason Wong Pierce Nichols Rick Berggreen Tri Le

Overview Background History Kinect development Kinect hardware Kinect features Expanding Kinect beyond video games

Kinect Technology Development Technology originally conceived in 2005 by Zeev Zalevsky, Alexander Shpunt, Aviad Maizels and Javier Garcia. Microsoft was not able to gain proprietary hold of the depth technology. PrimeSense now partners with the competitor of the Kinect for PC, ASUS Software design by Rare, a british game company and subsidiary of Microsoft Project Natal > Kinect

Kinect Processor Mystery Kinect originally included a processor in its specifications Sept 2010 Microsoft announces that the Kinect will not include the stated processor Released Kinect has a processor – Marvell PXA 168 Theories involve sharing resource demand with xBox, microphone spatial calculations, potential for further innovation

Inside the Kinect 1)Microhpone Array 2)IR Emitter 3)Depth Camera 4)Tilt Motor 5)USB Cable 6)Color Camera

Inside the Kinect

Structured Light Scanning

Kinect Light Pattern

Kinect Depth Map

Precision shape measurement, as of turbine blades Precision measurement of volume and planarity Precision measurement of cutting tool surface and structure Resolution on the order of 10 microns Structured Light Applications

Turbine Blade Gauging

Almost all non-Kinect structured light cameras are very expensive. The Kinect cuts the cost by not being a precision instrument – Low resolution – approximately minimum range. – Not much accuracy required to accurately reconstruct gestures etc. How did the Kinect Get to $150?

Features Skeletal Recognition and Tracking Beam Formation to identify the current sound source From microsoft sdk

Skeletal Recognition Use input depth map Infer body parts using a decision forest Produce joint position Positions are matched to predefined skeletal structures

Skeletal Recognition User captured depth image and synthetic motion capture data for training (object recognition base) 100k poses such that no two poses (max joint) are closer than 5 (cm)

Skeletal Recognition Body part / joint selection Correspond to the colors

Skeletal Recognition Per-pixel Depth Image extraction Randomized decision forest for each part

Skeletal Recognition Training parameters and classification accuracy

Joint position proposal Local mode-finding approach based on a mean shift with a weighted Gaussian kernel Each mode is therefore pushed back into the scene by a learned z offset zc to produce final joint position proposal

Skeletal Recognition Process repeats for each frame at 30Hz. The skeletal structure is tracked to differentiate different people in frame.

Audio Features Linear four-element microphone array Microphone array Improve echo cancellation Noise suppression Identify speaker through beam forming Microphone array characteristic: Noise gain and Directivity Index

Beam forming with Audio Array Figure 1. Microphone array directivity pattern in 3D, 1khz Figure 2: Nearly constant beam width in the diapason of 300 to 5kHz the microphone array engine searches for the speaker position and acts as if it points a beam at the current speaker. The higher directivity of the microphone array reduces the amount of captured ambient noises and reverberated waves.

Microphone Array Characteristics Microphone array Element s TypeNG, dBNGA, dBDI, dB Linear, small2uni-directional Linear, big2uni-directional Linear, 4el4uni-directional L-shaped4uni-directional Linear, 4 el second geometry 4integrated

Advances of Kinect Technology Unique, inexpensive, and compact design has led to new applications involving depth perception, video, and imagery Hobbyists take advantage of the depth camera to create automated moving objects Scientists incorporate motion sensing for hands-free assistance of pictures or remote tools

Advances of Kinect Technology Examples utilizing kinect: In Munich, students developed robotic car with Kinect to sent through obstacle courses In UK, team created robot to search through post-earthquake rubble for trapped victims Surgeons in Toronto now view critical patient images and x- rays in the surgery room hands-free instead of leaving room In Oxford, neurologist uses facial recognition and motion sensing to aid blind people with special glasses Professors at UW develop haptic rendering system to experience “remote touching” virtual environment

Bibliography people/multisite_files/mcp/imagecache/mcp2011_slideshow_large/mcp2011_images/alex-kipman-microsoft-i1.jpg