1. EEC-492/592 Kinect Application Development Lecture 10 Wenbing Zhao wenbing@ieee.org

2. Outline Human skeleton tracking (part III)

3. Skeleton Smoothing Skeleton data can be smoothed by registering a set of smooth parameters  As a parameter to SkeletonStream.Enable(….); // create the smooth parameters var smoothParameters = new TransformSmoothParameters { Correction = 0.1f, JitterRadius = 0.05f, MaxDeviationRadius = 0.05f, Prediction = 0.1f, Smoothing = 1.0f }; // Enable the skeleton stream with smooth parameters this.sensor.SkeletonStream.Enable(smoothParameters);

4. Smoothing Parameters Correction: specifies the amount of correction needed for the raw data. Range [0.0, 1.0]  The value must be within the range of 0 to 1.0 and the default value is 0.5.  With lower values more correction is applied, the raw data is corrected, and the data looks smoother Smoothing: determines the amount of smoothing applied while processing. Range [0.0, 1.0]  The value must be within the range of 0 to 1.0 and the default value is 0.5.  The larger value, the smoother skeleton data, however, it increases the latency  Zero value => you will get the raw skeleton data

5. Smoothing Parameters JitterRadius: limit the radius value for jittery data, >=0  Measured in meters and the default value is 0.5.  If the position of a jitter is outside the set radius, it is corrected to be positioned at the radius MaxDeviationRadius : max limit of the deviation that is allowed to be considered for determining a jitter, >=0  points outside of the MaxDeviationRadius range are not considered as jitter  default value is 0.04 meters Prediction: number of frames predicted into the future, >=0  default value: 0.0  A value greater than 0.5 will likely lead to overshoot when the data changes quickly

6. Kinect SDK Smoothing Algorithm Holt double exponential smoothing algorithm is used to reduce the jitters from skeletal joint data  The smoothing algorithm applies to each set of data and calculates a moving average based on the previous set of data  During the calculation of moving average, it uses the values passed by the smoothing parameter.

7. Getting Data Frames Together We have covered three (color, depth, skeleton) types of data streams that are returned by the sensor For a real application, we often need all three types of data streams We can use a single event AllFramesReady, which will do the job for all three of them The AllFramesReady event fires when new frames are available for the color, depth, and skeleton streams this.sensor.AllFramesReady+=sensor_AllFramesReady; void sensor_AllFramesReady(object sender, AllFramesReadyEventArgs e) { }

8. Getting Data Frames Together

9. Floor Determination Each skeleton frame contains a floor-clipping-plane vector, which contains the coefficients of an estimated floor-plane equation  The equation is normalized so that the physical interpretation of D is the height of the camera from the floor, in meters The skeleton tracking system updates this estimate for each frame and uses it as a clipping plane for removing the background and segmenting players Ax + By + Cz + D = 0

10. How to Use FloorClipPlane In Your App FloorClipPlane can be used to calculate the height of each joint with respect to the floor Can be used for validation study, as well as fall detection How to get FloorClipPlane parameters: How to calculate the height of a joint: float A = frame.FloorClipPlane.Item1; float B = frame.FloorClipPlane.Item2; float C = frame.FloorClipPlane.Item3; float D = frame.FloorClipPlane.Item4; float height = A*joint.Position.X+B*joint.Position.Y+C*joint.Position.Z+D;

11. Joint Orientation The bone orientation is provided in two forms:  A hierarchical rotation based on a bone relationship defined on the skeleton joint structure  An absolute orientation in Kinect camera coordinates The orientation information is provided in form of quaternions and rotation matrices for use in different animation scenarios BoneOrientation Class, public properties:  StartJoint: Gets the skeleton joint where the bone starts  EndJoint: Gets the skeleton joint where the bone ends  HierarchicalRotation: Gets or sets the rotation of a bone relative to its parent bone, of type BoneRotation  AbsoluteRotation: Gets or sets the rotation of the bone relative to camera coordinates, of type BoneRotation

12. BoneRotation Class BoneRotation has two properties  Matrix: Gets or sets a matrix representation of the bone rotation  Quaternion: Gets or sets a quaternion representation of the bone rotation Vector4 struct: has four properties  W, X, Y, Z Matrix4 struct: has 17 properties  Identity, of type Matrix4  M11, M12, M13, M14, M21, M22, M23, M24, M31, M32, M33, M34, M41, M42, M43, M44 public Matrix4 Matrix { get; set; } public Vector4 Quaternion { get; set; }

13. Access Joint Orientation private void DrawSkeletonsWithOrientations() { foreach (Skeleton skeleton in this.skeletonData) { if (skeleton.TrackingState == SkeletonTrackingState.Tracked) { foreach (BoneOrientation orientation in skeleton.BoneOrientations) { // Display bone with Rotation using quaternion DrawBonewithRotation(orientation.StartJoint, orientation.EndJoint, orientation.AbsoluteRotation.Quaternion); // Display hierarchical rotation using matrix DrawHierarchicalRotation(orientation.StartJoint, orientation.HierarchicalRotation.Matrix) }

14. Build ShapeGame App Create a new C# WPF project with name ShapeGame  It is a much simplified app from the Kinect ShapeGame  Only one shape, i.e., a ball is used  You are limited to use your righthand to hit the ball  Ball only drops from a designated spot down, i.e., no X velocity Add Microsoft.Kinect reference Design GUI Added WindowLoaded() method in xaml file Adding code

15. GUI Design Image control Canvas

16. Adding Code Add member variables: WindowLoaded():  Enable both ColorImageStream and SkeletonStream  Register event handler for both ColorFrameReady and SkeletonFrameReady events  Initialize the shape (i.e., ball) KinectSensor sensor; Skeleton[] totalSkeleton = new Skeleton[6]; WriteableBitmap colorBitmap; byte[] colorPixels; Skeleton skeleton; Thing thing = new Thing(); // a struct for ball double gravity = 0.017;

17. Adding Code The Thing Struct. Make it a private struct inside the MainWindow class private struct Thing { public System.Windows.Point Center; public double YVelocity; public double XVelocity; public Ellipse Shape; public bool Hit(System.Windows.Point joint) { double minDxSquared = this.Shape.RenderSize.Width; minDxSquared *= minDxSquared; double dist = SquaredDistance(Center.X, Center.Y, joint.X, joint.Y); if (dist<= minDxSquared) { return true; } else return false; }

18. Adding Code private static double SquaredDistance(double x1, double y1, double x2, double y2) { return ((x2 - x1) * (x2 - x1)) + ((y2 - y1) * (y2 - y1)); } private void WindowLoaded(object sender, RoutedEventArgs e) { // same as before …. // new code for ball initialization thing.Shape = new Ellipse(); thing.Shape.Width = 30; thing.Shape.Height = 30; thing.Shape.Fill = new SolidColorBrush(Color.FromRgb(0, 255, 255)); thing.Center.X = 300; thing.Center.Y = 0; thing.Shape.SetValue(Canvas.LeftProperty, thing.Center.X - thing.Shape.Width); thing.Shape.SetValue(Canvas.TopProperty, thing.Center.Y - thing.Shape.Width); canvas1.Children.Add(thing.Shape); }

19. Modify Event Handler for Skeleton Frames void skeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e) { canvas1.Children.Clear(); advanceThingPosition(); canvas1.Children.Add(thing.Shape); // remaining code same as before ….. Point handPt = ScalePosition(skeleton.Joints[JointType.HandRight].Position); if(thing.Hit(handPt)) { this.thing.YVelocity = -1.0*this.thing.YVelocity; }

20. Shape Animation Code void advanceThingPosition() { thing.Center.Offset(thing.XVelocity, thing.YVelocity); thing.YVelocity += this.gravity; thing.Shape.SetValue(Canvas.LeftProperty, thing.Center.X - thing.Shape.Width); thing.Shape.SetValue(Canvas.TopProperty, thing.Center.Y - thing.Shape.Width); // if goes out of bound, reset position, as well as velocity if (thing.Center.Y >= canvas1.Height) { thing.Center.Y = 0; thing.XVelocity = 0; thing.YVelocity = 0; }

21. Challenge Task For advanced students, improve the shape game in the following ways:  Keep the hit count, count should reset if ball falls to the bottom  Display the hit count at the ball  Consider the angle of the hit to the ball, and adjust the x velocity accordingly  Allow other joints to hit the ball too, such as left hand, shoulder, etc.  Add smoothing to skeleton data 9/3/2015 EEC492/693/793 - iPhone Application Development 21