1. 14th week, Applications
Hand Gesture Recognition and Virtual Game Control Based on 3D Accelerometer and EMG Sensors
Spring Semester, 2010

2. Outline Motivation Hand Gesture Recognition Virtual Game Control
Experiments
Summary

3. Hand Gesture-based HCI

4. Sensing Techniques Vision-based Movement-based EMG-based
＋ Easily track hand gestures
－ Sensitive to user’s circumstances
Movement-based
Glove: ＋ Achieve good performance / － wear a cumbersome glove (hinders the convenience and naturalness of HCI)
ACC: Easy to wear and recognize
EMG-based
Hands-free application yet less accurate

5. Motivation
Each sensing technique has its own advances and capabilities  multiple sensor fusion
ACC-based gesture control
Well suited to distinguish noticeable, larger scale gestures
EMG-based gesture control
Contain rich information about hand gestures of various size scales
We considered the complementary features of ACC (recognition) and EMG (segmentation + recognition)

6. Hand Gesture Recognition
Proposed system
Segmentation
Feature extraction
Recognition with HMM

7. Hand Gesture Recognition
Data Segmentation
As hand movement switches from one gesture to another one, the corresponding muscles relax for a while
Onset threshold
Determine active segments
Offset threshold
Prevent the fragmentation

8. Feature Extraction: EMG
Hand Gesture Recognition
Feature Extraction: EMG
Frame-based extraction
Every EMG channel is filtered by Hamming window
Minimize the signal discontinuities
4×n dimensional feature vectors
3rd order auto-regressive coefficients
Mean absolute value
Window size = 250ms

9. Feature Extraction: ACC
Hand Gesture Recognition
Feature Extraction: ACC
Normalize the variations in the scale and speed of gesture
Scale the amplitude of the data
Extrapolate the active segment to 32 points
3×32 dimensional feature vectors

10. HMM for Recognition Multi-stream HMMs Weight factors
Hand Gesture Recognition
HMM for Recognition
Multi-stream HMMs
Weight factors
Recognition result

11. Virtual Game Control

12. Experiments: Setup Delsys Myomonitor IV sensor system Five subjects
Four-channel EMG + a 3D-ACC
Five subjects
Utilize LR HMMs with five states
Built EMG and ACC HMMs independently (weight = 0.5)
Training set is collected by 10 repetitions

13. Experiments
Results

14. Summary
Hand gesture recognition can be utilized in natural interaction between human and computers
EMG + ACC to achieve real-time hand gesture recognition  Virtual Rubik’s Cube game