EEC-693/793 Applied Computer Vision with Depth Cameras Lecture 10 Wenbing Zhao wenbing@ieee.org 1

Outline Human skeleton tracking (part III)

Skeleton Smoothing Skeleton data can be smoothed by registering a set of smooth parameters As a parameter to SkeletonStream.Enable(….); // create the smooth parameters var smoothParameters = new TransformSmoothParameters { Correction = 0.1f, JitterRadius = 0.05f, MaxDeviationRadius = 0.05f, Prediction = 0.1f, Smoothing = 0.5f }; // Enable the skeleton stream with smooth parameters this.sensor.SkeletonStream.Enable(smoothParameters);

Smoothing Parameters Correction: specifies the amount of correction needed for the raw data. Range [0.0, 1.0] The value must be within the range of 0 to 1.0 and the default value is 0.5. Lower values are slower to correct towards the raw data and appear smoother, while higher values correct toward the raw data more quickly Smoothing: determines the amount of smoothing applied while processing. Range [0.0, 1.0] The larger value, the smoother skeleton data, however, it increases the latency Zero value => you will get the raw skeleton data

Smoothing Parameters JitterRadius: limit the radius value for jittery data, >=0 Measured in meters and the default value is 0.05. If the position of a jitter is outside the set radius, it is corrected to be positioned at the radius MaxDeviationRadius: max limit of the deviation that is allowed to be considered for determining a jitter, >=0 Filtered values that would exceed the radius from the raw data are clamped at this distance, in the direction of the filtered value default value is 0.04 meters Prediction: number of frames predicted into the future, >=0 default value: 0.5 A value greater than 0.5 will likely lead to overshoot when the data changes quickly

Kinect SDK Smoothing Algorithm Holt double exponential smoothing algorithm is used to reduce the jitters from skeletal joint data The smoothing algorithm applies to each set of data and calculates a moving average based on the previous set of data During the calculation of moving average, it uses the values passed by the smoothing parameter. Applying smoothing on skeleton data could be very expensive in terms of application performance. This is because the skeleton stream data itself is massive, and applying smoothing filtering on it makes data processing slow and this highly depends on the parameters you are using

Getting Data Frames Together We have covered three (color, depth, skeleton) types of data streams that are returned by the sensor For a real application, we often need all three types of data streams We can use a single event AllFramesReady, which will do the job for all three of them The AllFramesReady event fires when new frames are available for the color, depth, and skeleton streams this.sensor.AllFramesReady+=sensor_AllFramesReady; void sensor_AllFramesReady(object sender, AllFramesReadyEventArgs e) { }

Getting Data Frames Together

Floor Determination Each skeleton frame contains a floor-clipping-plane vector, which contains the coefficients of an estimated floor-plane equation The equation is normalized so that the physical interpretation of D is the height of the camera from the floor, in meters The skeleton tracking system updates this estimate for each frame and uses it as a clipping plane for removing the background and segmenting players Ax + By + Cz + D = 0

How to Use FloorClipPlane In Your App FloorClipPlane can be used to calculate the height of each joint with respect to the floor Can be used for validation study, as well as fall detection How to get FloorClipPlane parameters: How to calculate the height of a joint: float A = frame.FloorClipPlane.Item1; float B = frame.FloorClipPlane.Item2; float C = frame.FloorClipPlane.Item3; float D = frame.FloorClipPlane.Item4; float height = A*joint.Position.X+B*joint.Position.Y+C*joint.Position.Z+D;

Joint Orientation The bone orientation is provided in two forms: A hierarchical rotation based on a bone relationship defined on the skeleton joint structure An absolute orientation in Kinect camera coordinates The orientation information is provided in form of quaternions and rotation matrices for use in different animation scenarios BoneOrientation Class, public properties: StartJoint: Gets the skeleton joint where the bone starts EndJoint: Gets the skeleton joint where the bone ends HierarchicalRotation: Gets or sets the rotation of a bone relative to its parent bone, of type BoneRotation AbsoluteRotation: Gets or sets the rotation of the bone relative to camera coordinates, of type BoneRotation

BoneRotation Class BoneRotation has two properties Matrix: Gets or sets a matrix representation of the bone rotation Quaternion: Gets or sets a quaternion representation of the bone rotation Vector4 struct: has four properties W, X, Y, Z Matrix4 struct: has 17 properties Identity, of type Matrix4 M11, M12, M13, M14, M21, M22, M23, M24, M31, M32, M33, M34, M41, M42, M43, M44 public Matrix4 Matrix { get; set; } public Vector4 Quaternion { get; set; }

Access Joint Orientation private void DrawSkeletonsWithOrientations() { foreach (Skeleton skeleton in this.skeletonData) { if (skeleton.TrackingState == SkeletonTrackingState.Tracked) { foreach (BoneOrientation orientation in skeleton.BoneOrientations) { // Display bone with Rotation using quaternion DrawBonewithRotation(orientation.StartJoint, orientation.EndJoint, orientation.AbsoluteRotation.Quaternion); // Display hierarchical rotation using matrix DrawHierarchicalRotation(orientation.StartJoint, orientation.HierarchicalRotation.Matrix) }

Build ShapeGame App Create a new C# WPF project with name ShapeGame It is a much simplified app from the Kinect ShapeGame Only one shape, i.e., a ball is used You are limited to use your righthand to hit the ball Ball only drops from a designated spot down, i.e., no X velocity Add Microsoft.Kinect reference Design GUI Added WindowLoaded() method in xaml file Adding code

GUI Design Image control Canvas

Adding Code Add member variables: WindowLoaded(): Enable both ColorImageStream and SkeletonStream Register event handler for both ColorFrameReady and SkeletonFrameReady events Initialize the shape (i.e., ball) KinectSensor sensor; Skeleton[] totalSkeleton = new Skeleton[6]; WriteableBitmap colorBitmap; byte[] colorPixels; Skeleton skeleton; Thing thing = new Thing(); // a struct for ball double gravity = 0.017;

Adding Code The Thing Struct. Make it a private struct inside the MainWindow class private struct Thing { public System.Windows.Point Center; public double YVelocity; public double XVelocity; public Ellipse Shape; public bool Hit(System.Windows.Point joint) double minDxSquared = this.Shape.RenderSize.Width; minDxSquared *= minDxSquared; double dist = SquaredDistance(Center.X, Center.Y, joint.X, joint.Y); if (dist<= minDxSquared) return true; } else return false; }

Adding Code private static double SquaredDistance(double x1, double y1, double x2, double y2) { return ((x2 - x1) * (x2 - x1)) + ((y2 - y1) * (y2 - y1)); } private void WindowLoaded(object sender, RoutedEventArgs e) { // same as before …. // new code for ball initialization thing.Shape = new Ellipse(); thing.Shape.Width = 30; thing.Shape.Height = 30; thing.Shape.Fill = new SolidColorBrush(Color.FromRgb(0, 255, 255)); thing.Center.X = 300; thing.Center.Y = 0; thing.Shape.SetValue(Canvas.LeftProperty, thing.Center.X - thing.Shape.Width); thing.Shape.SetValue(Canvas.TopProperty, thing.Center.Y - thing.Shape.Width); canvas1.Children.Add(thing.Shape); }

Modify Event Handler for Skeleton Frames void skeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e) { canvas1.Children.Clear(); advanceThingPosition(); canvas1.Children.Add(thing.Shape); // remaining code same as before ….. Point handPt = ScalePosition(skeleton.Joints[JointType.HandRight].Position); if(thing.Hit(handPt)) this.thing.YVelocity = -1.0*this.thing.YVelocity; }

Shape Animation Code void advanceThingPosition() { thing.Center.Offset(thing.XVelocity, thing.YVelocity); thing.YVelocity += this.gravity; thing.Shape.SetValue(Canvas.LeftProperty, thing.Center.X - thing.Shape.Width); thing.Shape.SetValue(Canvas.TopProperty, thing.Center.Y - thing.Shape.Width); // if goes out of bound, reset position, as well as velocity if (thing.Center.Y >= canvas1.Height) thing.Center.Y = 0; thing.XVelocity = 0; thing.YVelocity = 0; }

EEC492/693/793 - iPhone Application Development Challenge Task For advanced students, improve the shape game in the following ways: Keep the hit count, count should reset if ball falls to the bottom Display the hit count at the ball Consider the angle of the hit to the ball, and adjust the x velocity accordingly Allow other joints to hit the ball too, such as left hand, shoulder, etc. Add smoothing to skeleton data 1/31/2019 EEC492/693/793 - iPhone Application Development