1. Massachusetts Institute of Technology
Gesture Recognition
Assaf Feldman
MASJ622
The Media Laboratory
Massachusetts Institute of Technology
Assaf Feldman

2. 11/12/2018
Introduction
ReachMedia is an on the move interactive system for delivering Just-in-Time information on every day objects
A wristband with an RFID reader detects objects that the user touches or holds (implicit input).
Assaf Feldman
Assaf Feldman

3. Supply information in an unobtrusive way. Requirments.
Challenges
Supply information in an unobtrusive way.
Requirments.
Hands free
Socially acceptable
Assaf Feldman

4. Current interfaces Too high tech
Prevent normal activities such flip book pages, make gestures or shake hands.
Cyber Glove
Assaf Feldman
Twiddler

5. Rely on existing hardware
Approach
Rely on existing hardware
Use the wristband as an explicit input method via gesture recognition.
Assaf Feldman

6. Sensor
Accelerometrs
Cheap
Small
Low power
Assaf Feldman

7. 3 axis acceleration (labeled by hand) ~100 samples per class
Data
3 axis acceleration (labeled by hand)
~100 samples per class
Classes
Noise
Assaf Feldman

8. Gesture Recognition Problem
“unknown” class consists of everything else.
High accuracy (small FP rate) require.
Preferably user independent (pre trained models) - Classifier must generalize over all exemplars of one class.
Assaf Feldman

9. Gesture Recognition Problem
“unknown” class consists of everything else.
High accuracy (small FP rate) required.
Preferably user independent (pre trained models) - Classifier must generalize over all exemplars of one class.
Assaf Feldman

10. Classification Algorithm Choice
Nature of Data
Training is possible
Real Time classification needed – can use raw data, no ffts, no convolutions
time series recognition techniques
Temporal data
Assaf Feldman

11. HMM performance w/wo noise
No noise - High accuracy between 3 gestures
With noise – Confusion (either FP or low accuracy)
Assaf Feldman

12. How Many? Choosing HMM model Bad result for uniform number of states
3 gestures are quite similar, big difference between 3 gestures and “unknown” class.
Need more states for “unknown” class
How Many?
Assaf Feldman

13. Choosing “unknown” class model
Fix number of states per gesture
Find evidence for “unknown class” with 10 fold cross validation
Repeat
Assaf Feldman

14. Results – “unknown class”
3 states HMM for gestures 5 states for “unknown” –tested on 30 samples per class
Assaf Feldman

15. Cons for “unknown” modeling
Easy to create a training interface for gestures - use a game or a small taks
How do you get a user to train “unknown” class
Assaf Feldman

16. Alternative – Use HMM outputs
Maybe the log probability outputs of the 3 models will be sufficient for creating a statistical rejection model
Method
Find HMM models for the 3 class problem
Use test data to collect log probability outputs for hits
Trains a GMM on the with the HMM outputs
Use cross validation to decide on a threshold for the distance
Assaf Feldman

17. Partial Results Distinctive clusters
Not separable (some noise in the middle of class 3
Need to test classifier (NB + GMM)
Assaf Feldman

18. Future Work
Finish testing binary claasification of HMM, maybe with some more time domain features added to the vector
Assaf Feldman