Body Tracking and Gesture Recognition Aaron Pulver M.A.C.S. Body Tracking and Gesture Recognition Aaron Pulver

Overview The M.A.C.S. needs to be able to sucessfully track users and recognize various hand gestures and/or body poses. Several sensor and software options were investigated to determine the most efficient combination

Related Marketing Requirements Description 1 Must autonomously follow an individual throughout a flat environment with obstacles. 2 Must adapt in height for different individuals. 3 Must identify an individual and follow only him or her. 4 Must maintain a reasonable distance from the individual. 5 Must respond to up to four hand gestures to aid in mobility.

Engineering Specification Engineering Specifications Related Marking Requirement Description 1 1,4 Must be able to move at up to 5 mph with a desired average speed of 3 mph while continuously tracking the user’s position. 2 4 The M.A.C.S. should be no more than two feet from the followed individual at any time. The Kinect/Sensor will be mounted on the back of the cart. 3 5 The M.A.C.S. should have an at least 65% accuracy when classifying gestures. The M.A.C.S. should identify the correct user by combining smartphone data with the Kinect sensor. Should the user be out of view, recalibration should occur within 3 seconds of seeing the person. The body tracking should track children (4ft) through large adults (6 ft-6”).

Risk Investigation (Hardware) Ease of use Availability Cost Webcam/Camera Difficult Yes $25-200 new Xtion Pro Moderate No (back-ordered) $179 new MS Kinect $109.99 new, $50 used (free for us for now)

Risk Investigation (Software) Ease of use Platforms Skeleton Tracking Languages OpenCV Difficult Linux, Windows No C++, Python, Java (wrappers for other languages) MS SDK Easy Windows Yes C#, VB.Net Libfreenect Moderate C,C++,C#, VB.net, Java OpenNI C++, (wrappers for .NET and Java)

Why the Kinect and OpenNI? Price and availability The Kinect has very similar attributes to the Xtion Pro but it is more widely used OpenNI has high-level libraries for skeleton tracking OpenNI is multi-platform and multi-language OpenNI is actively supported and developed by PrimeSense OpenNI has good documentation and examples

Tracking Risk Mitigation Mount the Kinect high on the cart in the back Skeleton Tracking Use accelerometer data from phone Communicate to user if lost Tracking Module Motor Controller Proximity Sensors Smartphone user data Kinect location of user(s) Joint(s) Warning Message/Light Gesture Recognition

Kinect Placement Kinect Proximity Sensors 43°

Gesture Recognition Risk Mitigation Skeleton tracking of joint positions/rotations Machine learning of gestures Support Vector Machine Dynamic Time Warping LIBSVM – open source SVM library Cross-validation to verify correct learning Raw Kinect Sensor Data OpenNI More Preprocessing LIBSVM and Classification Controller

Parts List Part Description Cost Our Cost Availability MS Kinect (Xbox 360) A depth, image, and IR sensor. $109.99 $50.00 (used) $40.00 (used) Amazon Newegg Craigslist (used) Android Smart Phone Smartphone to stream accelerometer data. $199.99 Free N/A Laptop (Linux) A laptop to process the Kinect data. $299.99

Testing Strategy (Tracking) User Walking down a hallway Turn Left Walk Straight Turn Left, then Left again (around a wall or something) Requires a robot that moves Multiple users in the field of view Track user  Another user enters  Users cross paths several times  Users obstruct each other for 3 seconds  Initial user leave FOV  Initial user returns to FOV Initial user leaves FOV for too long must recalibrate with gesture

Testing Strategy (Gestures) The gesture recognition testing is embedded in the machine learning process Cross-validation If the recorded test data meets or exceeds the 65% classification rate then it is successful Real-time testing can also be done but it should yield similar results to the test data.

Uncertainties Gesture recognition Time and effort Not as critical as other pieces of the project Data collection and the machine learning can be done while the drive base is being built

Sources [1] Browning, R. C., Baker, E. A., Herron, J. A. and Kram, R. (2006). "Effects of obesity and sex on the energetic cost and preferred speed of walking". Journal of Applied Physiology 100 (2): 390– 398.   [2] Hall, Edward T. (1966). The Hidden Dimension. Anchor Books. [3] “Kinect for Windows Sensor Components and Specifications”. Microsoft Store. Microsoft. Retrieved October 30, 2013. http://msdn.microsoft.com/en-us/library/jj131033.aspx [4] Bhattacharya, S.; Czejdo, B.; Perez, N., "Gesture classification with machine learning using Kinect sensor data," Emerging Applications of Information Technology (EAIT), 2012 Third International Conference on , vol., no., pp.348,351, Nov. 30 2012-Dec. 1 2012 [5] Bodiroza, S.; Doisy, G.; Hafner, V.V., "Position-invariant, real-time gesture recognition based on dynamic time warping," Human-Robot Interaction (HRI), 2013 8th ACM/IEEE International Conference on , vol., no., pp.87,88, 3-6 March 2013 [6] “Xtion PRO”. Asus. Asus. Retrieved October 31, 2013. http://www.asus.com/Multimedia/Xtion_PRO/#specifications

Questions?