Massachusetts Institute of Technology Gesture Recognition Assaf Feldman MASJ622 The Media Laboratory Massachusetts Institute of Technology Assaf Feldman

11/12/2018 Introduction ReachMedia is an on the move interactive system for delivering Just-in-Time information on every day objects A wristband with an RFID reader detects objects that the user touches or holds (implicit input). Assaf Feldman Assaf Feldman

Supply information in an unobtrusive way. Requirments. Challenges Supply information in an unobtrusive way. Requirments. Hands free Socially acceptable Assaf Feldman

Current interfaces Too high tech Prevent normal activities such flip book pages, make gestures or shake hands. Cyber Glove Assaf Feldman Twiddler

Rely on existing hardware Approach Rely on existing hardware Use the wristband as an explicit input method via gesture recognition. Assaf Feldman

Sensor Accelerometrs Cheap Small Low power Assaf Feldman

3 axis acceleration (labeled by hand) ~100 samples per class Data 3 axis acceleration (labeled by hand) ~100 samples per class Classes Noise Assaf Feldman

Gesture Recognition Problem “unknown” class consists of everything else. High accuracy (small FP rate) require. Preferably user independent (pre trained models) - Classifier must generalize over all exemplars of one class. Assaf Feldman

Gesture Recognition Problem “unknown” class consists of everything else. High accuracy (small FP rate) required. Preferably user independent (pre trained models) - Classifier must generalize over all exemplars of one class. Assaf Feldman

Classification Algorithm Choice Nature of Data Training is possible Real Time classification needed – can use raw data, no ffts, no convolutions time series recognition techniques Temporal data Assaf Feldman

HMM performance w/wo noise No noise - High accuracy between 3 gestures   With noise – Confusion (either FP or low accuracy) Assaf Feldman

How Many? Choosing HMM model Bad result for uniform number of states 3 gestures are quite similar, big difference between 3 gestures and “unknown” class. Need more states for “unknown” class How Many? Assaf Feldman

Choosing “unknown” class model Fix number of states per gesture Find evidence for “unknown class” with 10 fold cross validation Repeat Assaf Feldman

Results – “unknown class” 3 states HMM for gestures 5 states for “unknown” –tested on 30 samples per class Assaf Feldman

Cons for “unknown” modeling Easy to create a training interface for gestures - use a game or a small taks How do you get a user to train “unknown” class Assaf Feldman

Alternative – Use HMM outputs Maybe the log probability outputs of the 3 models will be sufficient for creating a statistical rejection model Method Find HMM models for the 3 class problem Use test data to collect log probability outputs for hits Trains a GMM on the with the HMM outputs Use cross validation to decide on a threshold for the distance Assaf Feldman

Partial Results Distinctive clusters Not separable (some noise in the middle of class 3 Need to test classifier (NB + GMM) Assaf Feldman

Future Work Finish testing binary claasification of HMM, maybe with some more time domain features added to the vector Assaf Feldman