Introduction to Microsoft Kinect Sensor Programming

Slides:



Advertisements
Similar presentations
Page 1 | Microsoft Work With Depth Data Kinect for Windows Video Courses Jan 2013.
Advertisements

What’s New in Kinect for Windows v2 Click to add title
Capturing Your Audience with Kinect
Joshua Fabian Tyler Young James C. Peyton Jones Garrett M. Clayton Integrating the Microsoft Kinect With Simulink: Real-Time Object Tracking Example (
KINECT Vinayak Thapliyal and Noah Balsmeyer 1. Overview  What is the Kinect?  Why was it made?  How does it work?  How does it compare to other sensors?
Natural User Interface with Kinect for Windows Clemente Giorio & Paolo Patierno.
Wait, what? More than just technology catch-up. Johnny Lee (Carnegie Mellon) * Motion-Tracking/Head-Tracking/Virtual Whiteboard
By : Adham Suwan Mohammed Zaza Ahmed Mafarjeh. Achieving Security through Kinect using Skeleton Analysis (ASKSA)
5/19/2015 EEC484/584: Computer Networks 1 EEC-490 Senior Design (CE) Kinect Programming Tutorial 1 Wenbing Zhao
ALFRED THOMPSON MICROSOFT ACADEMIC TEAM Kinect for FRC 2012.
Work With Skeleton Data
Kinect SDK Tutorial Skeleton and Camera (RGB) Anant Bhardwaj.
CPVR 2013 Tutorial. Native Managed Applications Toolkit Drivers Runtime Skeletal Tracking.
By Rishabh Maheshwari. Objective of today’s lecture Play Angry Birds in 3D.
SM1205 Interactivity Topic 01: Introduction Spring 2012SCM-CityU1.
Game Development with Kinect
SM1205 Interactivity Topic 01: Introduction Spring 2011SCM-CityU1.
Page 1 | Microsoft Work With Color Data Kinect for Windows Video Courses Jan 2013.
1 References: 1. J.M. Hart, Windows System Programming, 4th Ed., Addison-Wesley, 2010, Ch.12 2.Microsoft Kinect SDK for Developers,
Page 1 | Microsoft Introduction to audio stream Kinect for Windows Video Courses.
Page 1 | Microsoft Streams sync and coordinate mapping Kinect for Windows Video Courses.
Graphics and Multimedia In visual Studio. Net (C#)
EEC-492/592 Kinect Application Development Lecture 10 Wenbing Zhao
Kinect Part II Anna Loparev.
Professor : Yih-Ran Sheu Student’s name : Nguyen Van Binh Student ID: MA02B203 Kinect camera 1 Southern Taiwan University Department of Electrical Engineering.
Kinect SDK Crash Course (In 12 slides or less) Elliot Babchick.
INTERACTING WITH SIMULATION ENVIRONMENTS THROUGH THE KINECT Fayez Alazmi Supervisor: Dr. Brett Wilkinson Flinders University Image 1Image 2 Source : 1.
Page 1 | Microsoft Work With Skeleton Data Kinect for Windows Video Courses Jan 2013.
Page 1 | Microsoft Work With Color Data Kinect for Windows Video Courses Jan 2013.
Coding4Fun: Build Fun, Cool, Commercial Applications Using the Kinect for Windows SDK Dan Fernandez Director Microsoft Corporation Rick Barraza Senior.
Project By: Brent Elder, Mike Holovka, Hisham Algadaibi.
1 EEC-492/592 Kinect Application Development Lecture 2 Wenbing Zhao
Project By: Brent Elder, Mike Holovka, Hisham Algadaibi.
Programming with the Kinect for Windows SDK
Connect with your. Hi I’m Justin Weinberg I’m a Carnegie Mellon Graduate Manager Consultant at Sogeti Founder of the North Texas Silverlight User Group.
Professor : Tsung Fu Chien Student’s name : Nguyen Trong Tuyen Student ID: MA02B208 An application Kinect camera controls Vehicles by Gesture 1 Southern.
12/5/2015 EEC492/693/793 - iPhone Application Development 1 EEC-693/793 Applied Computer Vision with Depth Cameras Lecture 4 Wenbing Zhao
CONTENT 1. Introduction to Kinect 2. Some Libraries for Kinect 3. Implement 4. Conclusion & Future works 1.
KINECT FOR WINDOWS Ken Casada Developer Evangelist, Microsoft Switzerland | blogblog.
Coding4Fun: Build Fun, Cool, Commercial Applications Using the Kinect for Windows SDK Dan Fernandez Director Microsoft Corporation Rick Barraza Senior.
Introduction to Kinect For Windows SDK
INO301. BGR32 Format – Every Pixel (0,0 | 0,1 | 0,2) has blue, green, red, empty BGREmpty Kinect Image Sizes: 80x60, 320x240, 640x480.
EEC-693/793 Applied Computer Vision with Depth Cameras
EEC-693/793 Applied Computer Vision with Depth Cameras Lecture 8 Wenbing Zhao
2/16/2016 EEC492/693/793 - iPhone Application Development 1 EEC-693/793 Applied Computer Vision with Depth Cameras Lecture 4 Wenbing Zhao
Coding4Fun: Build Fun, Cool, Commercial Applications Using the Kinect for Windows SDK Dan Fernandez Director Microsoft Corporation Brian Peek Senior Technical.
EEC-693/793 Applied Computer Vision with Depth Cameras Lecture 9 Wenbing Zhao
Expressive Intelligence Studio // Center for Games and Playable Media // Microsoft Kinect Intro CMPS179 Game.
1/64. 2/64 3/64 4/64 5/64 6/64 7/64 8/64 9/64.
Creative Coding & the New Kinect
Southern Taiwan University Department of Electrical Engineering
5/5/2018 1:12 AM © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS.
EEC-693/793 Applied Computer Vision with Depth Cameras
EEC-693/793 Applied Computer Vision with Depth Cameras
Programming HCI Yingcai Xiao Yingcai Xiao.
EEC-693/793 Applied Computer Vision with Depth Cameras
EEC-693/793 Applied Computer Vision with Depth Cameras
EEC-693/793 Applied Computer Vision with Depth Cameras
EEC-693/793 Applied Computer Vision with Depth Cameras
Kinect for Creative Development with open source frameworks
EEC-492/592 Kinect Application Development
EEC-693/793 Applied Computer Vision with Depth Cameras
EEC-693/793 Applied Computer Vision with Depth Cameras
EEC-693/793 Applied Computer Vision with Depth Cameras
EEC-693/793 Applied Computer Vision with Depth Cameras
EEC-693/793 Applied Computer Vision with Depth Cameras
EEC-693/793 Applied Computer Vision with Depth Cameras
EEC-693/793 Applied Computer Vision with Depth Cameras
EEC-693/793 Applied Computer Vision with Depth Cameras
EEC-693/793 Applied Computer Vision with Depth Cameras
Presentation transcript:

Introduction to Microsoft Kinect Sensor Programming Presented by: Mahmoud Awad Microsoft PhD Scholar

What is Kinect Sensor Full Skeleton Tracking Gesture Recognition Intuitive Interactive Interface Voice Recognition Capabilities Affordable Easy to install

What is Kinect Sensor RGB Camera – A regular video camera that has a resolution of 640x480 at 30 frames per second 3D Depth Sensors – This is a combination of a depth projector (left reticule) and sensor (right reticule) to calculate distance. Motorized Tilt – The motorized tilt enables you to adjust the camera up or down 27 degrees to ensure that the camera has the best possible view Multi-Array Microphone – A four-microphone array that is mounted as a single microphone

What is Kinect Sensor

Installing Kinect Sensor Download the 32-bit or 64-bit version of the Kinect for Windows SDK at http://kinectforwindows.org Visual Studio 2010 (Professional or Express edition) DirectX DirectX® SDK - June 2010 or later version Current runtime for Microsoft DirectX® 9

Installing Kinect Sensor For Speech Recognition: Microsoft Speech Platform Runtime, version 10.2, we suggest that you install both the 32-bit and 64-bit runtime. Microsoft Speech Platform - Software Development Kit, version 10.2 – select 32-bit or 64-bit according to your Windows installation Kinect for Windows Runtime Language Pack, version 0.9

Kinect SDK Fundamentals Start New Windows Presentation Foundation Project Add a reference to: Mucrosoft.Research.Kinect Add “Using” statement using Microsoft.Research.Kinect.Nui; using Microsoft.Research.Kinect.Audio;

Kinect SDK Fundamentals Initializing the runtime in the  Window_Loaded event  if (Runtime.Kinects.Count == 0)     {         this.Title = "No Kinect connected";     }     else         //use first Kinect         nui = Runtime.Kinects[0];         nui.Initialize(RuntimeOptions.UseColor | RuntimeOptions.UseDepthAndPlayerIndex | RuntimeOptions.UseSkeletalTracking);

Kinect SDK Fundamentals We can: Use more than one kinect but;  can only have one Kinect do skeletal tracking at a time. Track kinect status changed: Connected Disconnected Not powered Not ready error

Setting Runtime Options in the Window_Loaded event, initialize the runtime with the options you want to use. For example, set RuntimeOptions.UseColor to use the RGB camera and Depth Data: nui.Initialize(RuntimeOptions.UseColor | RuntimeOptions.UseDepth); The example below sets the runtime to use the color camera, a depth camera, and skeletal tracking: nui.Initialize(RuntimeOptions.UseColor | RuntimeOptions.UseDepthAndPlayerIndex | RuntimeOptions.UseSkeletalTracking );

Uninitializing the Runtime Remember that when you are done using the Kinect Runtime, you should call the Uninitialize method nui.Uninitialize();

Kinect Events Register Events then open streams: RGB Camera:  VideoFrameReady Depth Camera: DepthFrameReady Tracked Skeleton: SkeletonFrameReady Open a video or a depth stream.

RGB Camera Event nui.VideoFrameReady += new EventHandler<ImageFrameReadyEventArgs>(nui_VideoFrameReady); //hit tab twice void nui_VideoFrameReady(object sender, ImageFrameReadyEventArgs e) {    // event code goes here }

video frame ready returned values

Converting the byte[] array to an image PlanarImage imageData = e.ImageFrame.Image; image1.Source = BitmapSource.Create(imageData.Width, imageData.Height, 96, 96, PixelFormats.Bgr32, null, imageData.Bits, data.Width * imageData.BytesPerPixel);

Open Streams Video stream: nui.VideoStream.Open(ImageStreamType.Video, 2, ImageResolution.Resolution640x480, ImageType.Color); Depth Stream:  nui.DepthStream.Open(ImageStreamType.Depth, 2 ,ImageResolution.Resolution320x240,  ImageType.DepthAndPlayerIndex);

Calculating Distance There are 2 bytes per pixel. Use bitshift operator to get the distance for a particular pixel. private int GetDistanceWithPlayerIndex(byte firstFrame, byte secondFrame) {   //offset by 3 in first byte to get value after player index   int distance = (int)(firstFrame >> 3 | secondFrame << 5);   return distance; }

Skeleton Tracking Runtime object: Initialize Runtime: Register Event: Runtime nui = new Runtime(); Initialize Runtime: nui.Initialize(RuntimeOptions.UseSkeletalTracking); Register Event: nui.SkeletonFrameReady += new EventHandler<SkeletonFrameReadyEventArgs>(nui_SkeletonFrameReady);

Skeleton Tracking In event handler get all tracking data: void nui_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e) {     SkeletonFrame allSkeletons = e.SkeletonFrame;       //get the first tracked skeleton     SkeletonData skeleton = (from s in allSkeletons.Skeletons                              where s.TrackingState == SkeletonTrackingState.Tracked                              select s).FirstOrDefault(); }

Joint Positions A Joint position returns X,Y,Z values as explained below X = Horizontal position measured as the distance, in meters from the Kinect along the X Axis Y = Vertical position measured as the distance, in meters from the Kinect along the Y Axis Z = Distance from Kinect measured in meters

Getting head position for example: skeleton.Joints[JointID.Head].Position.X skeleton.Joints[JointID.Head].Position.Y Getting Left hand position: skeleton.Joints[JointID.HandLeft].Position.X skeleton.Joints[JointID.HandLeft].Position.Y

We can scale the values to maximum x and Y values. Use ScaleTo method from coding4fun kinect toolkit. Joint HandRight =  skeleton.Joints[JointID.HandRight].ScaleTo(640, 480);

Using TransformSmoothing to for less skeletal jitter Use  TransformSmooth property. TransformSmoothParameters parameters = new TransformSmoothParameters();          parameters.Smoothing = 0.7f;          parameters.Correction = 0.3f;          parameters.Prediction = 0.4f;          parameters.JitterRadius = 1.0f;          parameters.MaxDeviationRadius = 0.5f;          nui.SkeletonEngine.SmoothParameters = parameters;

For more information http://www.microsoft.com/en-us/kinectforwindows/ http://www.i-programmer.info/programming/hardware/2623-getting-started-with-microsoft-kinect-sdk.html http://channel9.msdn.com/Series/KinectSDKQuickstarts