1. Connect with your

2. Hi I’m Justin Weinberg I’m a Carnegie Mellon Graduate Manager Consultant at Sogeti Founder of the North Texas Silverlight User Group

11. Wow!

12. Motorized Tilt RGB Camera 3D Depth Sensors Microphone Array

13. That’s not USB? You’ll need this..

14. http://openkinect.org/ http://www.openni.org/ http://research.microsoft.com/kinectsdk/

15. http:// research.microsoft.com/kinect / http://www.microsoft.com/express

16. Color Camera Streams Depth / Depth with Player Streams Skeletal Tracking Audio Functionality http:// research.microsoft.com/kinect /

17. Demo 0 Hello Tilt

18. Kinect NUI Big Picture Runtime nui = new Runtime(); nui.VideoFrameReady +=... nui.VideoStream.Open(...) void nui_VideoFrameReady(object sender, ImageFrameReadyEventArgs e) { //Process e.ImageFrame. }

19. 1 2 3 4 9 10 11 12 876 5 13 14 15 16 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 ImageFrame.Image.Bits With camera data you might expect Byte[,] What you actually get is Byte[]:

20. BGR32 Format width in pixels height in pixels Stride: Bytes Per Row Bytes per pixel * Width 4 Bytes Per pixel Byte Array:

21. Demo 1 Hello Colorful World

22. Depth data 500mm 2000mm 2030mm 2010mm 2000mm 500mm 2500mm

23. Depth data Max resolution for raw depth data is 640 x 480 2 bytes per pixel of depth data distance(0,0) = byte0 | byte1 << 8 11011110 pixel 00001010 11101010 00000101 11101010 00001110 11011110 00001010

24. Demo 2 Depth

25. Depth + Player Data 500mm 2000mm 2030mm 2010mm 2000mm 500mm 2500mm player 1

26. Depth + Player Data Max resolution is 320 x 240 11011001 00001011 Player: byte0 & 7 00000001 00000111 11011001 & Depth(0,0) = byte0 >> 3 | byte1 << 5 Still 2 bytes per pixel But player is last 3 bits in first byte 1101100001011001

27. Depth + Player Index distance In mm 11 bits per pixel distance In mm distance In mm 11011001 XXXX0111 player distance In mm player distance In mm player distance In mm player distance In mm player distance In mm player distance In mm distance In mm player distance In mm player distance In mm player 000 = No player 001 = Player 1 010 = Player 2 11011001 XXXX0101 11011001 XXXX0111

28. Demo 3 Depth + Player Data

29. Skeletal Tracking

30. skeletonFrame.Skeletons where skeleton.TrackingState == SkeletonTrackingState.Tracked skeletonFrame.Skeletons from skeleton in skeletonFrame.Skeletons where skeleton.TrackingState == SkeletonTrackingState.PositionOnly

31. hand_left wrist_left elbow_left shoulder_left hand_right wrist_right elbow_right shoulder_right hip_left head hip_right knee_left knee_right ankle_right ankle_left foot_leftfoot_right hip center spine shoulder_center Skeleton.Joints[JointId enum] Joint.Position (X,Y,Z,W) Skeletons are mirrored X, Y, Z in left handed cartesian system +Z is out from sensor. W always returns 1 at this moment.

32. Demo 3 Skeletal Tracking

33. Demo 4 Putting Visuals Together

34. Kinect Audio

36. Kinect Audio API Microphone Array DMO Implementation Beamforming and Sound Localization Features

37. Beam Forming

38. -0.875 radians -50 degrees 0.875 radians 50 degrees.0175 radians 10 degrees

39. MicArrayMode MicArrayAdaptiveBeamAdaptive beam forming. Raises beamforming events MicArrayExternBeamYour app chooses a beam (0 -10), May be broke. Use with MicArrayBeamAngle. MicArrayFixedBeamAlways use the center beam. MicArraySimpleSumAll inputs are used (No beamforming) MicArraySingleBeamPerform beam forming. An internal source localizer selects the beam. MicArraySingleChanOne input is used. Set last 8 bits to choose the input source. SoundSourcePosition SoundSourcePositionConfidence BeamChanged Event

40. Accousting Echo Cancellation

41. AEC bool NoiseFill // Clip removes residual audio but leaves gaps. This fills with noise if true bool CenterClip // After AEC, some echo residuals will remain. Setting to true removes. int EchoLength – Sets an adaptive filter to the AEC algorithm (in MS). Recommended values are 128, 256, 512, 1024. For most offices and homes, use 256ms int FrameSize // Size of a frame in samples Allowed values are : 80, 128, 160, 240, 256, 320. System default for microphone array only is 320 All other modes default is 160 int SpeakerIndex – Device index of speaker apply AEC process to

42. Noise Suppression

43. Noise Supression bool NoiseSuppression – True enables, false disables.

44. Automatic Gain Control

45. bool AutomaticGainControl – True enables, false disables.

46. Demo 5 Recording Audio

47. Audio @!%!#!! Gotchas Use Microsoft.Speech not System.Speech Must run in an MTA AEC SystemModes crash if no speaker output. Must plug Kinect into Wall DevicePairGuid / Qualitymetrics = crash Should run in ThreadMode Priority Highest

48. Demo 6 Speech recognition

49. Demo 7 Kinect Effect (Thanks SoulSolutions!)

50. Creative Commons Acknowledgments Flickr by Ian Lewis Flickr by major nelson Flickr by markodesign Flickr by stephantom Flickr by Abstract Machine Flickr By Aixo.co.kr Flickr By Andresmh Flickr by Antony Bennison Flickr by Ars Electronica Flickr By bm.iphone Flickr by campuspartymexico Flickr by Chris Jagers Flickr By ci_polla Paola Sucato Flickr by creepyhalloweenimages Flickr by cvander Flickr by IMAL.org Flickr by Institut Douglas Flickr by Joshua and AmberFlickr by Logon Sakai Flickr by openexhibits Flickr by Paul_Clarke Flickr by petyr.rahl Flickr by pt Flickr by Stilldavid Flickr by y3rdua Flickr by yesyesnono Flickr by yum9me Flickr by Jeff Howard Flickr by jhscreenwm Flickr by Kyle McDonald Flickr by Morten Rand-Hendriksen Flickr by Stuart Herbert Flickr by Visiophone Flickr by orkomedix Michael Zöllner and Stephan Huber at the University of Konstanz Openclip by J_Alves Openclip by jetxee Openclip by molumen Openclip by shokunin Openclip by Jhnri Openclip by Machovka Adafruit image of Hector Martin Thanks!!