EEC-693/793 Applied Computer Vision with Depth Cameras Lecture 4 Wenbing Zhao wenbing@ieee.org 2/23/2019 EEC492/693/793 - iPhone Application Development 1 1

Outline Working with Kinect color stream Types of color images Retrieving color images: event & polling models Steps on handling of color images Event handler: C# delegate Build the KinectCam app 2 2

Working with Kinect Streams Kinect SDK supports two image stream formats Color image stream Depth image stream Both are children of ImageStream class The image frames are stored into a buffer for use by the application. If there is any delay in reading the buffer data and rendering it as images, the buffer will be overwritten => old frame lost Main steps in handling image streams Enabling the stream Capturing the stream frame by frame Processing the image frames 3

Types of Color Images RGB: read-green-blue color space (RGB) YUV: Each pixel is an array of four: Blue, Green, Red, Alpha Alpha: transparency degree 32bits per pixel at 640x480 at 30FPS or 1280x960 at 12FPS YUV: Y: luminance channel; U: blue channel, V: red channel 16 bits per pixel at 640x480 at 15 FPS Uses less memory than RGB Bayer: raw Bayer color image format with a Bayer color filter array (or Bayer filter): 50% green, 25% red, 25% blue Resolution at 640x480 at 30 FPS or 1280x960 at 12 FPS Infrared data: 16 bits per pixel at 640x480 at 30 FPS

Retrieving Color Image Stream Two ways: event model and polling model Event model Kinect sends the frame to the app whenever a new frame is captured by the sensor Need to register a event handler for the event

Polling Model The application sends a request to the sensor whenever there is a need to get an image frame Pass the time interval after which the sensor will return the image frame

Capturing Color Images Main steps Starting Kinect sensor Enable color stream channel with desirable image format Subscribe (register) an event handler (a method) with the stream channel Process the image frame Render the image frame on the user interface

Enable Color Stream Channel In class ColorImageStream, two methods public void Enable(); By default, use RgbResolution640x480Fps30 public void Enable(ColorImageFormat format); RgbResolution640x480Fps30 RgbResolution1280x960Fps12 YuvResolution640x480Fps15 RawYuvResolution640x480Fps15 InfraredResolution640x480Fps30 RawBayerResolution640x480Fps30 RawBayerResolution1280x960Fps12 Undefined You can enable only one type of color stream at a time

Registering an Event Handler this.sensor.ColorFrameReady += colorFrameReady; What is colorFrameReady?! It is the event handler that you will implement You can use any method name as you wish The method must have the following signature: void colorFrameReady(object sender, ColorImageFrameReadyEventArgs e) Sender: the object that fires the event e: the object that holds data to be retrieved (i.e., image frame)

Event Handler Event handler is similar to a function pointer in C/C++ In C# (and objective C), the event handler is a delegate with pedefined signature General format: public delegate void MyEventHandler(object sender, MyEventArgs e); Sender: files the event e: holds the data to be handled A delegate allows you to pass methods of one class to objects of other classes that can call those methods

Delegate in C# (http://www. developerfusion using System;  // Step 1. Declare a delegate with the signature of the encapsulated method  public delegate void MyDelegate(string input);  //Step 2. Define methods that match with signature of delegate declaration  class MyClass1{  public void delegateMethod1(string input){         Console.WriteLine("delegateMethod1 and input to the method is {0}",input);     }     public void delegateMethod2(string input){         Console.WriteLine("delegateMethod2 and input to the method is {0}",input);     }  }  }

Delegate in C# (http://www. developerfusion //Step 3. Create delegate object and plug in the methods  class MyClass2 {     public MyDelegate createDelegate(){         MyClass1 c2=new MyClass1();         MyDelegate d1 = new MyDelegate(c2.delegateMethod1);         MyDelegate d2 = new MyDelegate(c2.delegateMethod2);         MyDelegate d3 = d1 + d2;         return d3;     }  } 

Delegate in C# (http://www. developerfusion //Step 4. Call the encapsulated methods through the delegate  class MyClass3{     public void callDelegate(MyDelegate d, string input){         d(input);     }  }  class Driver{     static void Main(string[] args){         MyClass2 c2 = new MyClass2();         MyDelegate d = c2.createDelegate();         MyClass3 c3 = new MyClass3();         c3.callDelegate(d,"Calling the delegate");     }  }

Event Handler Delegate for ColorImage namespace System { public delegate void EventHandler<TEventArgs> (object sender, TEventArgs e); } In Kinect.Sensor class That is why we can this: public event EventHandler<ColorImageFrameReadyEventArgs> ColorFrameReady; this.sensor.ColorFrameReady += colorFrameReady;

Retrieve the Color Image Frame void colorFrameReady(object sender, ColorImageFrameReadyEventArgs e) { using (ColorImageFrame imageFrame = e.OpenColorImageFrame()) ……. } What is using(a new object) { }? It defines the scope of the new object Once the last statement in the {} block is executed, the object is garbage deleted

Handling/Displaying Color Image // copy image to a byte array imageFrame.CopyPixelDataTo(colorPixels); // width in bytes of a single row of pixel data including padding. int stride = imageFrame.Width * imageFrame.BytesPerPixel; // Write the pixel data into our bitmap this.colorBitmap.WritePixels( new Int32Rect(0, 0, this.colorBitmap.PixelWidth, this.colorBitmap.PixelHeight), this.colorPixels, stride, 0); // Don’t do this for stride calculation // If format is changed, it may not work! this.colorBitmap.PixelWidth * sizeof(int);

Color Image Handling public void WritePixels(Int32Rect srcRect, Array pixels, int stride, int offset); Updates the pixels in the specified region of the bitmap. srcRect: The rectangle of the System.Windows.Media.Imaging.WriteableBitmap to update pixels: The pixel array used to update the bitmap. stride: The stride of the update region in pixels. offset: The input buffer offset

WriteableBitmap Class Inherits from BitmapSource Constructor pixelWidth:width of the bitmap pixelHeight: height of the bitmap dpiX: horizontal dots per inch (dpi) dpiY: vertical dots per inch pixelFormat: pixel format palette: finite set of colors public WriteableBitmap(int pixelWidth, int pixelHeight, double dpiX, double dpiY, PixelFormat pixelFormat, BitmapPalette palette);

System.Windows.Media.PixelFormats Bgr32 property: blue, green, red channels (3 bytes), last byte is set to 0 http://msdn.microsoft.com/en-us/magazine/cc534995.aspx Bgra32 Bgr24 Gray2 Gray16 Etc.

Build KinectCam App Create a new WFP project named KinectCam Draw the GUI for the app Modify xaml file to add WindowLoaded and WindowClosing as before Add member variables for displaying color images Start Kinect sensor, enable color stream, register color stream handler, connect image control to the bitmap for display Process each color image

Design GUI Add label title Add image control

Add Member Variables KinectSensor sensor; // Bitmap that will hold color information private WriteableBitmap colorBitmap; // Intermediate storage for the color data received from the camera private byte[] colorPixels;

Initialization (in WindowLoaded()) if (KinectSensor.KinectSensors.Count > 0) { this.sensor = KinectSensor.KinectSensors[0]; if (this.sensor != null && !this.sensor.IsRunning) { this.sensor.ColorStream.Enable(); this.colorPixels = new byte[this.sensor.ColorStream.FramePixelDataLength]; this.colorBitmap = new WriteableBitmap(this.sensor.ColorStream.FrameWidth, this.sensor.ColorStream.FrameHeight, 96.0, 96.0, PixelFormats.Bgr32, null); this.image1.Source = this.colorBitmap; this.sensor.ColorFrameReady += this.colorFrameReady; this.sensor.Start(); } else { MessageBox.Show("No device is connected!"); this.Close(); }

Handling/Displaying Color Images void colorFrameReady(object sender, ColorImageFrameReadyEventArgs e) { using (ColorImageFrame imageFrame = e.OpenColorImageFrame()) { if (null == imageFrame) return; imageFrame.CopyPixelDataTo(colorPixels); int stride = imageFrame.Width * imageFrame.BytesPerPixel; // Write the pixel data into our bitmap this.colorBitmap.WritePixels( new Int32Rect(0, 0, this.colorBitmap.PixelWidth, this.colorBitmap.PixelHeight), this.colorPixels, stride, 0); }