Introduction to Microsoft Kinect Sensor Programming Presented by: Mahmoud Awad Microsoft PhD Scholar

What is Kinect Sensor Full Skeleton Tracking Gesture Recognition Intuitive Interactive Interface Voice Recognition Capabilities Affordable Easy to install

What is Kinect Sensor RGB Camera – A regular video camera that has a resolution of 640x480 at 30 frames per second 3D Depth Sensors – This is a combination of a depth projector (left reticule) and sensor (right reticule) to calculate distance. Motorized Tilt – The motorized tilt enables you to adjust the camera up or down 27 degrees to ensure that the camera has the best possible view Multi-Array Microphone – A four-microphone array that is mounted as a single microphone

What is Kinect Sensor

Installing Kinect Sensor Download the 32-bit or 64-bit version of the Kinect for Windows SDK at http://kinectforwindows.org Visual Studio 2010 (Professional or Express edition) DirectX DirectX® SDK - June 2010 or later version Current runtime for Microsoft DirectX® 9

Installing Kinect Sensor For Speech Recognition: Microsoft Speech Platform Runtime, version 10.2, we suggest that you install both the 32-bit and 64-bit runtime. Microsoft Speech Platform - Software Development Kit, version 10.2 – select 32-bit or 64-bit according to your Windows installation Kinect for Windows Runtime Language Pack, version 0.9

Kinect SDK Fundamentals Start New Windows Presentation Foundation Project Add a reference to: Mucrosoft.Research.Kinect Add “Using” statement using Microsoft.Research.Kinect.Nui; using Microsoft.Research.Kinect.Audio;

Kinect SDK Fundamentals Initializing the runtime in the  Window_Loaded event  if (Runtime.Kinects.Count == 0)     {         this.Title = "No Kinect connected";     }     else         //use first Kinect         nui = Runtime.Kinects[0];         nui.Initialize(RuntimeOptions.UseColor | RuntimeOptions.UseDepthAndPlayerIndex | RuntimeOptions.UseSkeletalTracking);

Kinect SDK Fundamentals We can: Use more than one kinect but;  can only have one Kinect do skeletal tracking at a time. Track kinect status changed: Connected Disconnected Not powered Not ready error

Setting Runtime Options in the Window_Loaded event, initialize the runtime with the options you want to use. For example, set RuntimeOptions.UseColor to use the RGB camera and Depth Data: nui.Initialize(RuntimeOptions.UseColor | RuntimeOptions.UseDepth); The example below sets the runtime to use the color camera, a depth camera, and skeletal tracking: nui.Initialize(RuntimeOptions.UseColor | RuntimeOptions.UseDepthAndPlayerIndex | RuntimeOptions.UseSkeletalTracking );

Uninitializing the Runtime Remember that when you are done using the Kinect Runtime, you should call the Uninitialize method nui.Uninitialize();

Kinect Events Register Events then open streams: RGB Camera:  VideoFrameReady Depth Camera: DepthFrameReady Tracked Skeleton: SkeletonFrameReady Open a video or a depth stream.

RGB Camera Event nui.VideoFrameReady += new EventHandler<ImageFrameReadyEventArgs>(nui_VideoFrameReady); //hit tab twice void nui_VideoFrameReady(object sender, ImageFrameReadyEventArgs e) {    // event code goes here }

video frame ready returned values

Converting the byte[] array to an image PlanarImage imageData = e.ImageFrame.Image; image1.Source = BitmapSource.Create(imageData.Width, imageData.Height, 96, 96, PixelFormats.Bgr32, null, imageData.Bits, data.Width * imageData.BytesPerPixel);

Open Streams Video stream: nui.VideoStream.Open(ImageStreamType.Video, 2, ImageResolution.Resolution640x480, ImageType.Color); Depth Stream:  nui.DepthStream.Open(ImageStreamType.Depth, 2 ,ImageResolution.Resolution320x240,  ImageType.DepthAndPlayerIndex);

Calculating Distance There are 2 bytes per pixel. Use bitshift operator to get the distance for a particular pixel. private int GetDistanceWithPlayerIndex(byte firstFrame, byte secondFrame) {   //offset by 3 in first byte to get value after player index   int distance = (int)(firstFrame >> 3 | secondFrame << 5);   return distance; }

Skeleton Tracking Runtime object: Initialize Runtime: Register Event: Runtime nui = new Runtime(); Initialize Runtime: nui.Initialize(RuntimeOptions.UseSkeletalTracking); Register Event: nui.SkeletonFrameReady += new EventHandler<SkeletonFrameReadyEventArgs>(nui_SkeletonFrameReady);

Skeleton Tracking In event handler get all tracking data: void nui_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e) {     SkeletonFrame allSkeletons = e.SkeletonFrame;       //get the first tracked skeleton     SkeletonData skeleton = (from s in allSkeletons.Skeletons                              where s.TrackingState == SkeletonTrackingState.Tracked                              select s).FirstOrDefault(); }

Joint Positions A Joint position returns X,Y,Z values as explained below X = Horizontal position measured as the distance, in meters from the Kinect along the X Axis Y = Vertical position measured as the distance, in meters from the Kinect along the Y Axis Z = Distance from Kinect measured in meters

Getting head position for example: skeleton.Joints[JointID.Head].Position.X skeleton.Joints[JointID.Head].Position.Y Getting Left hand position: skeleton.Joints[JointID.HandLeft].Position.X skeleton.Joints[JointID.HandLeft].Position.Y

We can scale the values to maximum x and Y values. Use ScaleTo method from coding4fun kinect toolkit. Joint HandRight =  skeleton.Joints[JointID.HandRight].ScaleTo(640, 480);

Using TransformSmoothing to for less skeletal jitter Use  TransformSmooth property. TransformSmoothParameters parameters = new TransformSmoothParameters();          parameters.Smoothing = 0.7f;          parameters.Correction = 0.3f;          parameters.Prediction = 0.4f;          parameters.JitterRadius = 1.0f;          parameters.MaxDeviationRadius = 0.5f;          nui.SkeletonEngine.SmoothParameters = parameters;

For more information http://www.microsoft.com/en-us/kinectforwindows/ http://www.i-programmer.info/programming/hardware/2623-getting-started-with-microsoft-kinect-sdk.html http://channel9.msdn.com/Series/KinectSDKQuickstarts