1. Body Tracking and Gesture Recognition Aaron Pulver
M.A.C.S.
Body Tracking and Gesture Recognition
Aaron Pulver

2. Overview
The M.A.C.S. needs to be able to sucessfully track users and recognize various hand gestures and/or body poses.
Several sensor and software options were investigated to determine the most efficient combination

3. Related Marketing Requirements
Description 1
Must autonomously follow an individual throughout a flat environment with obstacles. 2
Must adapt in height for different individuals. 3
Must identify an individual and follow only him or her. 4
Must maintain a reasonable distance from the individual. 5
Must respond to up to four hand gestures to aid in mobility.

4. Engineering Specification
Engineering Specifications
Related Marking Requirement
Description 1
1,4
Must be able to move at up to 5 mph with a desired average speed of 3 mph while continuously tracking the user’s position. 2 4
The M.A.C.S. should be no more than two feet from the followed individual at any time. The Kinect/Sensor will be mounted on the back of the cart. 3 5
The M.A.C.S. should have an at least 65% accuracy when classifying gestures.
The M.A.C.S. should identify the correct user by combining smartphone data with the Kinect sensor. Should the user be out of view, recalibration should occur within 3 seconds of seeing the person.
The body tracking should track children (4ft) through large adults (6 ft-6”).

5. Risk Investigation (Hardware)
Ease of use
Availability
Cost
Webcam/Camera
Difficult
Yes
$ new
Xtion Pro
Moderate
No (back-ordered)
$179 new
MS Kinect
$ new, $50 used (free for us for now)

6. Risk Investigation (Software)
Ease of use
Platforms
Skeleton Tracking
Languages
OpenCV
Difficult
Linux, Windows No
C++, Python, Java (wrappers for other languages)
MS SDK
Easy
Windows
Yes
C#, VB.Net
Libfreenect
Moderate
C,C++,C#, VB.net, Java
OpenNI
C++, (wrappers for .NET and Java)

7. Why the Kinect and OpenNI?
Price and availability
The Kinect has very similar attributes to the Xtion Pro but it is more widely used
OpenNI has high-level libraries for skeleton tracking
OpenNI is multi-platform and multi-language
OpenNI is actively supported and developed by PrimeSense
OpenNI has good documentation and examples

8. Tracking Risk Mitigation
Mount the Kinect high on the cart in the back
Skeleton Tracking
Use accelerometer data from phone
Communicate to user if lost
Tracking Module
Motor Controller
Proximity Sensors
Smartphone user data
Kinect location of user(s) Joint(s)
Warning Message/Light
Gesture Recognition

9. Kinect Placement
Kinect
Proximity Sensors
43°

10. Gesture Recognition Risk Mitigation
Skeleton tracking of joint positions/rotations
Machine learning of gestures
Support Vector Machine
Dynamic Time Warping
LIBSVM – open source SVM library
Cross-validation to verify correct learning
Raw Kinect Sensor Data
OpenNI
More Preprocessing
LIBSVM and
Classification
Controller

11. Parts List Part Description Cost Our Cost Availability
MS Kinect (Xbox 360)
A depth, image, and IR sensor.
$109.99
$50.00 (used)
$ (used)
Amazon
Newegg
Craigslist (used)
Android Smart Phone
Smartphone to stream accelerometer data.
$199.99
Free
N/A
Laptop (Linux)
A laptop to process the Kinect data.
$299.99

12. Testing Strategy (Tracking)
User Walking down a hallway
Turn Left Walk Straight Turn Left, then Left again (around a wall or something)
Requires a robot that moves
Multiple users in the field of view
Track user  Another user enters  Users cross paths several times  Users obstruct each other for 3 seconds  Initial user leave FOV  Initial user returns to FOV
Initial user leaves FOV for too long must recalibrate with gesture

13. Testing Strategy (Gestures)
The gesture recognition testing is embedded in the machine learning process
Cross-validation
If the recorded test data meets or exceeds the 65% classification rate then it is successful
Real-time testing can also be done but it should yield similar results to the test data.

14. Uncertainties Gesture recognition Time and effort
Not as critical as other pieces of the project
Data collection and the machine learning can be done while the drive base is being built

15. Sources
[1] Browning, R. C., Baker, E. A., Herron, J. A. and Kram, R. (2006). "Effects of obesity and sex on the energetic cost and preferred speed of walking". Journal of Applied Physiology 100 (2): 390– 398.
[2] Hall, Edward T. (1966). The Hidden Dimension. Anchor Books.
[3] “Kinect for Windows Sensor Components and Specifications”. Microsoft Store. Microsoft. Retrieved October 30,
[4] Bhattacharya, S.; Czejdo, B.; Perez, N., "Gesture classification with machine learning using Kinect sensor data," Emerging Applications of Information Technology (EAIT), 2012 Third International Conference on , vol., no., pp.348,351, Nov Dec
[5] Bodiroza, S.; Doisy, G.; Hafner, V.V., "Position-invariant, real-time gesture recognition based on dynamic time warping," Human-Robot Interaction (HRI), th ACM/IEEE International Conference on , vol., no., pp.87,88, 3-6 March 2013
[6] “Xtion PRO”. Asus. Asus. Retrieved October 31,

16. Questions?