1. EEC-693/793 Applied Computer Vision with Depth Cameras
Lecture 10
Wenbing Zhao 1

2. Outline
Human skeleton tracking (part III)

3. Skeleton Smoothing
Skeleton data can be smoothed by registering a set of smooth parameters
As a parameter to SkeletonStream.Enable(….);
// create the smooth parameters
var smoothParameters = new TransformSmoothParameters {
Correction = 0.1f,
JitterRadius = 0.05f,
MaxDeviationRadius = 0.05f,
Prediction = 0.1f,
Smoothing = 0.5f };
// Enable the skeleton stream with smooth parameters
this.sensor.SkeletonStream.Enable(smoothParameters);

4. Smoothing Parameters
Correction: specifies the amount of correction needed for the raw data. Range [0.0, 1.0]
The value must be within the range of 0 to 1.0 and the default value is 0.5.
Lower values are slower to correct towards the raw data and appear smoother, while higher values correct toward the raw data more quickly
Smoothing: determines the amount of smoothing applied while processing. Range [0.0, 1.0]
The larger value, the smoother skeleton data, however, it increases the latency
Zero value => you will get the raw skeleton data

5. Smoothing Parameters
JitterRadius: limit the radius value for jittery data, >=0
Measured in meters and the default value is 0.05.
If the position of a jitter is outside the set radius, it is corrected to be positioned at the radius
MaxDeviationRadius: The maximum radius in meters that filtered positions are allowed to deviate from raw data, >=0
Filtered values that would exceed the radius from the raw data are clamped at this distance, in the direction of the filtered value
default value is 0.04 meters
Prediction: number of frames predicted into the future, >=0
default value: 0.5
A value greater than 0.5 will likely lead to overshoot when the data changes quickly

6. Kinect SDK Smoothing Algorithm
Holt double exponential smoothing algorithm is used to reduce the jitters from skeletal joint data
The smoothing algorithm applies to each set of data and calculates a moving average based on the previous set of data
During the calculation of moving average, it uses the values passed by the smoothing parameter.
Applying smoothing on skeleton data could be very expensive in terms of application performance. This is because the skeleton stream data itself is massive, and applying smoothing filtering on it makes data processing slow and this highly depends on the parameters you are using

7. Getting Data Frames Together
We have covered three (color, depth, skeleton) types of data streams that are returned by the sensor
For a real application, we often need all three types of data streams
We can use a single event AllFramesReady, which will do the job for all three of them
The AllFramesReady event fires when new frames are available for the color, depth, and skeleton streams
this.sensor.AllFramesReady+=sensor_AllFramesReady;
void sensor_AllFramesReady(object sender, AllFramesReadyEventArgs e) { }

8. Getting Data Frames Together

9. Floor Determination
Each skeleton frame contains a floor-clipping-plane vector, which contains the coefficients of an estimated floor-plane equation
The equation is normalized so that the physical interpretation of D is the height of the camera from the floor, in meters
The skeleton tracking system updates this estimate for each frame and uses it as a clipping plane for removing the background and segmenting players
Ax + By + Cz + D = 0

10. How to Use FloorClipPlane In Your App
FloorClipPlane can be used to calculate the height of each joint with respect to the floor
Can be used for validation study, as well as fall detection
How to get FloorClipPlane parameters:
How to calculate the height of a joint:
float A = frame.FloorClipPlane.Item1;
float B = frame.FloorClipPlane.Item2;
float C = frame.FloorClipPlane.Item3;
float D = frame.FloorClipPlane.Item4;
float height = A*joint.Position.X+B*joint.Position.Y+C*joint.Position.Z+D;

11. Joint Orientation The bone orientation is provided in two forms:
A hierarchical rotation based on a bone relationship defined on the skeleton joint structure
An absolute orientation in Kinect camera coordinates
The orientation information is provided in form of quaternions and rotation matrices for use in different animation scenarios
BoneOrientation Class, public properties:
StartJoint: Gets the skeleton joint where the bone starts
EndJoint: Gets the skeleton joint where the bone ends
HierarchicalRotation: Gets or sets the rotation of a bone relative to its parent bone, of type BoneRotation
AbsoluteRotation: Gets or sets the rotation of the bone relative to camera coordinates, of type BoneRotation

12. BoneRotation Class BoneRotation has two properties
Matrix: Gets or sets a matrix representation of the bone rotation
Quaternion: Gets or sets a quaternion representation of the bone rotation
Vector4 struct: has four properties
W, X, Y, Z
Matrix4 struct: has 17 properties
Identity, of type Matrix4
M11, M12, M13, M14, M21, M22, M23, M24, M31, M32, M33, M34, M41, M42, M43, M44
public Matrix4 Matrix { get; set; }
public Vector4 Quaternion { get; set; }

13. Access Joint Orientation
private void DrawSkeletonsWithOrientations() {
foreach (Skeleton skeleton in this.skeletonData) {
if (skeleton.TrackingState == SkeletonTrackingState.Tracked) {
foreach (BoneOrientation orientation in skeleton.BoneOrientations)
{ // Display bone with Rotation using quaternion
DrawBonewithRotation(orientation.StartJoint, orientation.EndJoint,
orientation.AbsoluteRotation.Quaternion);
// Display hierarchical rotation using matrix
DrawHierarchicalRotation(orientation.StartJoint,
orientation.HierarchicalRotation.Matrix) }

14. Build ShapeGame App Create a new C# WPF project with name ShapeGame
It is a much simplified app from the Kinect ShapeGame
Only one shape, i.e., a ball is used
You are limited to use your righthand to hit the ball
Ball only drops from a designated spot down, i.e., no X velocity
Add Microsoft.Kinect reference
Design GUI
Added WindowLoaded() method in xaml file
Adding code

15. GUI Design
Image control
Canvas

16. Adding Code Add member variables: WindowLoaded():
Enable both ColorImageStream and SkeletonStream
Register event handler for both ColorFrameReady and SkeletonFrameReady events
Initialize the shape (i.e., ball)
KinectSensor sensor;
Skeleton[] totalSkeleton = new Skeleton[6];
WriteableBitmap colorBitmap;
byte[] colorPixels;
Skeleton skeleton;
Thing thing = new Thing(); // a struct for ball
double gravity = 0.017;

17. Adding Code
The Thing Struct. Make it a private struct inside the MainWindow class
private struct Thing {
public System.Windows.Point Center;
public double YVelocity;
public double XVelocity;
public Ellipse Shape;
public bool Hit(System.Windows.Point joint)
double minDxSquared = this.Shape.RenderSize.Width;
minDxSquared *= minDxSquared;
double dist = SquaredDistance(Center.X, Center.Y, joint.X, joint.Y);
if (dist<= minDxSquared)
return true;
} else
return false; }

18. Adding Code
private static double SquaredDistance(double x1, double y1, double x2, double y2) {
return ((x2 - x1) * (x2 - x1)) + ((y2 - y1) * (y2 - y1)); }
private void WindowLoaded(object sender, RoutedEventArgs e) {
// same as before ….
// new code for ball initialization
thing.Shape = new Ellipse();
thing.Shape.Width = 30; thing.Shape.Height = 30;
thing.Shape.Fill = new SolidColorBrush(Color.FromRgb(0, 255, 255));
thing.Center.X = 300; thing.Center.Y = 0;
thing.Shape.SetValue(Canvas.LeftProperty, thing.Center.X - thing.Shape.Width);
thing.Shape.SetValue(Canvas.TopProperty, thing.Center.Y - thing.Shape.Width);
canvas1.Children.Add(thing.Shape); }
Canvas.LeftProperty: A Double that represents the offset position from the left side of a parent Canvas.

19. Modify Event Handler for Skeleton Frames
void skeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e) {
canvas1.Children.Clear();
advanceThingPosition();
canvas1.Children.Add(thing.Shape);
// remaining code same as before
…..
Point handPt = ScalePosition(skeleton.Joints[JointType.HandRight].Position);
if(thing.Hit(handPt))
this.thing.YVelocity = -1.0*this.thing.YVelocity; }

20. Shape Animation Code void advanceThingPosition() {
thing.Center.Offset(thing.XVelocity, thing.YVelocity);
thing.YVelocity += this.gravity;
thing.Shape.SetValue(Canvas.LeftProperty, thing.Center.X - thing.Shape.Width);
thing.Shape.SetValue(Canvas.TopProperty, thing.Center.Y - thing.Shape.Width);
// if goes out of bound, reset position, as well as velocity
if (thing.Center.Y >= canvas1.Height)
thing.Center.Y = 0;
thing.XVelocity = 0;
thing.YVelocity = 0; }

21. EEC492/693/793 - iPhone Application Development
Challenge Task
For advanced students, improve the shape game in the following ways:
Keep the hit count, count should reset if ball falls to the bottom
Display the hit count at the ball
Consider the angle of the hit to the ball, and adjust the x velocity accordingly (you must consider physics here)
Allow other joints to hit the ball too, such as left hand, shoulder, etc.
Add smoothing to skeleton data
Add user guidance on his/her position based ClippedEdges
4/30/2019
EEC492/693/793 - iPhone Application Development