1. EEC-693/793 Applied Computer Vision with Depth Cameras
Lecture 9
Wenbing Zhao 1

2. Outline
Human skeleton tracking (part II)

3. Skeleton Tracking in Near Mode
Must enable near mode for DepthStream and near mode tracking for SkeletonStream
By default EnableTrackingInNearRange is set to false
this.sensor.DepthStream.Range = DepthRange.Near;
this.sensor.SkeletonStream.EnableTrackingInNearRange = true;

4. The Skeleton Class
The Skeleton class has a property named ClippedEdges, which is of type
FrameEdges, that describes which parts of the skeleton are out of the Kinect's
view.
They indicate which portion of the body is
getting cut off from the Kinect sensor view area

5. Choosing Which Skeleton to Track
Each skeleton is identified by a unique integer identifer: TrackingId
The TrackingId for a tracked skeleton is positive (>0)
The TrackingId of the skeleton that was tracked last should always be greater than the previously tracked skeleton’s Id
We can use TrackingId to specify which skeletons are to be tracked or used by the application
This will ensure that the skeleton tracking engine tracks only the skeleton
that you have identified and ignores the others.
sensor.SkeletonStream.AppChoosesSkeletons = true;
sensor.SkeletonStream.ChooseSkeletons(skeleton.TrackingId);

6. Monitoring Changes in the Skeleton
To track the development of a particular user, it is important to tell if the user has gone out of view and another user comes into view
The TrackingId can be used to monitor the changes in the skeleton being tracked (assuming only one user in view)
Use a local variable to store the TrackingId for the current skeleton
Update it when the skeleton ID is changed
public int CurrentSkeletonID =0;
if (skeleton != null && this.CurrentSkeletonID != skeleton.TrackingId) {
this.CurrentSkeletonID = skeleton.TrackingId; }

7. Monitoring Changes in the Skeleton: A Use Case
Intrusion detector app: in skeleton frame event handler
Set TrackingId when a new skeleton is detected, and save an image, only track the skeleton
Continue tracking this skeleton without saving more images
When the current skeleton is no longer tracked, reset to the default tracking mode (SDK tracks all possible skeletons in view)
public int CurrentSkeletonID =0; // member variable in MainApp class

8. Monitoring Changes in the Skeleton: A Use Case
Skeleton skeleton; // local variable in event handler
If(CurrentSkeletonID == 0) {
skeleton = (from trackSkeleton in totalSkeleton
where trackSkeleton.TrackingState == SkeletonTrackingState.Tracked
select trackSkeleton).FirstOrDefault();
if (skeleton == null) {
return; }
CurrentSkeletonID = skeleton.TrackingId;
this.sensor.SkeletonStream.AppChoosesSkeletons = true;
this.sensor.SkeletonStream.ChooseSkeletons(CurrentSkeletonID);
if (skeleton.Joints[JointType.Head].TrackingState ==
JointTrackingState.Tracked) {
this.SaveImage();

9. Monitoring Changes in the Skeleton: A Use Case
if (CurrentSkeletonID != 0) // we are currently tracking a skeleton {
skeleton = (from trackSkeleton in totalSkeleton
where trackSkeleton.TrackingState == SkeletonTrackingState.Tracked
&& trackSkeleton.TrackingId == CurrentSkeletonID
select trackSkeleton).FirstOrDefault();
if (skeleton == null)
// reset to default tracking mode
CurrentSkeletonID = 0;
this.sensor.SkeletonStream.AppChoosesSkeletons = false; }

10. Joints and JointCollection
Each Skeleton object has a property named Joints, which is a type of JointCollection and contains all the traceable joints
JointCollecton contains a set of Joints and can be accessed by specifying the index value
When you pass JointType to get the Joint point, it will return the Joint object
Skeleton skeleton = (from trackskeleton in totalSkeleton
where trackskeleton.TrackingState == SkeletonTrackingState.Tracked
select trackskeleton).FirstOrDefault();
Joint headJoint= skeleton.Joints[JointType.Head]);

11. Joint The Joint structure has three properties JointType: 20 joints
Positions: SkeletonPoint type (x,y,z)
TrackingState: NotTracked, Tracked, Inferred

12. Bones: Connecting Joints
Bones are the visual representation between joints
The complete hierarchy of a skeleton is composed of a series of bones
Each bone in a skeleton hierarchy has a parent joint and a child joint
Every joint can be a parent and child joint unless it is a leaf joint

13. Bone Sequence for a Default Skeleton
Root joint: Highest joint in hierarchy. Each skeleton has one root joint
For a default skeleton: Hip Center is the root joint
We can move and orient the entire skeleton in the skeleton space by translating
and rotating the root joint

14. Bone Sequence for a Seated Skeleton
For seated skeleton: Shoulder Center is the root joint

15. Drawing bones between joints
Bones are the visual representation between joints and can be represented by a line or any other object
void drawBone(Joint trackedJoint1, Joint trackedJoint2) {
Line skeletonBone = new Line();
skeletonBone.Stroke = Brushes.Black;
skeletonBone.StrokeThickness = 3;
Point joint1 = this.ScalePosition(trackedJoint1.Position);
skeletonBone.X1 = joint1.X; skeletonBone.Y1 = joint1.Y;
Point joint2 = this.ScalePosition(trackedJoint2.Position);
skeletonBone.X2 = joint2.X; skeletonBone.Y2 = joint2.Y;
myCanvas.Children.Add(skeletonBone); }

16. ClippedEdges
ClippedEdges (of type FrameEdges) describes which parts of the skeleton are out of the Kinect's view:
None, Right, Left, Top, Bottom
Can be used to give user feedbacks on the position

17. Build TrackingSkeleton App
Create a new C# WPF project with name TrackingSkeleton
Add Microsoft.Kinect reference
Design GUI
Added WindowLoaded() method in xaml file
Adding code

18. GUI Design
Image control
Canvas
TextBlock

19. Adding Code Add member variables: WindowLoaded():
Enable both ColorImageStream and SkeletonStream
Register event handler for both ColorFrameReady and SkeletonFrameReady events
Make sure to use default skeleton tracking mode (not seated mode!)
KinectSensor sensor;
Skeleton[] totalSkeleton = new Skeleton[6];
WriteableBitmap colorBitmap;
byte[] colorPixels;
Skeleton skeleton;
int currentSkeletonID = 0;

20. Add Event Handler for skeleton frames
void skeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e) {
canvas1.Children.Clear();
using (SkeletonFrame skeletonFrame = e.OpenSkeletonFrame()) {
if (skeletonFrame == null) { return; }
skeletonFrame.CopySkeletonDataTo(totalSkeleton);
skeleton = (from trackskeleton in totalSkeleton
where trackskeleton.TrackingState == SkeletonTrackingState.Tracked
select trackskeleton).FirstOrDefault();
if (skeleton == null)
return;
if (skeleton != null && this.currentSkeletonID != skeleton.TrackingId) {
this.currentSkeletonID = skeleton.TrackingId;
int totalTrackedJoints = skeleton.Joints.Where(item => item.TrackingState == JointTrackingState.Tracked).Count();
string TrackedTime = DateTime.Now.ToString("hh:mm:ss");
string status = "Skeleton Id: " + this.currentSkeletonID + ", total tracked joints: " + totalTrackedJoints + ", TrackTime: " + TrackedTime+"\n";
this.textBlock1.Text += status; }
DrawSkeleton(skeleton); }}

21. Adding Code
private void DrawSkeleton(Skeleton skeleton) {
drawBone(skeleton.Joints[JointType.Head], skeleton.Joints[JointType.ShoulderCenter]);
drawBone(skeleton.Joints[JointType.ShoulderCenter], skeleton.Joints[JointType.Spine]);
drawBone(skeleton.Joints[JointType.ShoulderCenter], skeleton.Joints[JointType.ShoulderLeft]);
drawBone(skeleton.Joints[JointType.ShoulderLeft], skeleton.Joints[JointType.ElbowLeft]);
drawBone(skeleton.Joints[JointType.ElbowLeft], skeleton.Joints[JointType.WristLeft]);
drawBone(skeleton.Joints[JointType.WristLeft], skeleton.Joints[JointType.HandLeft]);
drawBone(skeleton.Joints[JointType.ShoulderCenter], skeleton.Joints[JointType.ShoulderRight]);
drawBone(skeleton.Joints[JointType.ShoulderRight], skeleton.Joints[JointType.ElbowRight]);
drawBone(skeleton.Joints[JointType.ElbowRight], skeleton.Joints[JointType.WristRight]);
drawBone(skeleton.Joints[JointType.WristRight], skeleton.Joints[JointType.HandRight]);
// more on next slide }

22. Adding Code private Point ScalePosition(SkeletonPoint skeletonPoint) {
private void DrawSkeleton(Skeleton skeleton) {
// continue from previous slid
drawBone(skeleton.Joints[JointType.Spine], skeleton.Joints[JointType.HipCenter]);
drawBone(skeleton.Joints[JointType.HipCenter], skeleton.Joints[JointType.HipLeft]);
drawBone(skeleton.Joints[JointType.HipLeft], skeleton.Joints[JointType.KneeLeft]);
drawBone(skeleton.Joints[JointType.KneeLeft], skeleton.Joints[JointType.AnkleLeft]);
drawBone(skeleton.Joints[JointType.AnkleLeft], skeleton.Joints[JointType.FootLeft]);
drawBone(skeleton.Joints[JointType.HipCenter], skeleton.Joints[JointType.HipRight]);
drawBone(skeleton.Joints[JointType.HipRight], skeleton.Joints[JointType.KneeRight]);
drawBone(skeleton.Joints[JointType.KneeRight], skeleton.Joints[JointType.AnkleRight]);
drawBone(skeleton.Joints[JointType.AnkleRight], skeleton.Joints[JointType.FootRight]); }
private Point ScalePosition(SkeletonPoint skeletonPoint) {
DepthImagePoint depthPoint = this.sensor.CoordinateMapper.MapSkeletonPointToDepthPoint(skeletonPoint, DepthImageFormat.Resolution320x240Fps30);
return new Point(depthPoint.X, depthPoint.Y); }

23. Adding Code void drawBone(Joint trackedJoint1, Joint trackedJoint2) {
Line bone = new Line();
bone.Stroke = Brushes.Red;
bone.StrokeThickness = 3;
Point joint1 = this.ScalePosition(trackedJoint1.Position);
bone.X1 = joint1.X;
bone.Y1 = joint1.Y;
Point joint2 = this.ScalePosition(trackedJoint2.Position);
bone.X2 = joint2.X;
bone.Y2 = joint2.Y;
canvas1.Children.Add(bone); }

24. EEC492/693/793 - iPhone Application Development
Challenge Task
For advanced students, measure the angle formed between left/right arm and torso
Draw an arc between the arm and torso
Display the angle value with a TextBox on top of the arc
12/6/2018
EEC492/693/793 - iPhone Application Development

25. Helper Code for Advanced Task (https://dynamicdatadisplay. codeplex
private void AddCircularArcGraph(Point startPoint, Point endPoint, Size size) {
PathFigure pf = new PathFigure();
pf.StartPoint = new Point(startPoint.X, startPoint.Y);
ArcSegment arcSegment = new ArcSegment();
arcSegment.Point = new Point(endPoint.X, endPoint.Y);
arcSegment.Size = size;
arcSegment.SweepDirection = SweepDirection.Counterclockwise;
PathSegmentCollection psc = new PathSegmentCollection();
psc.Add(arcSegment); pf.Segments = psc;
PathFigureCollection pfc = new PathFigureCollection(); pfc.Add(pf);
PathGeometry pg = new PathGeometry(); pg.Figures = pfc;
var path = new System.Windows.Shapes.Path();
path.Stroke = Brushes.Black;
path.StrokeThickness = 1; path.Data = pg;
canvas1.Children.Add(path); }