14th week, Applications Hand Gesture Recognition and Virtual Game Control Based on 3D Accelerometer and EMG Sensors Spring Semester, 2010

Outline Motivation Hand Gesture Recognition Virtual Game Control Experiments Summary

Hand Gesture-based HCI

Sensing Techniques Vision-based Movement-based EMG-based ＋ Easily track hand gestures － Sensitive to user’s circumstances Movement-based Glove: ＋ Achieve good performance / － wear a cumbersome glove (hinders the convenience and naturalness of HCI) ACC: Easy to wear and recognize EMG-based Hands-free application yet less accurate

Motivation Each sensing technique has its own advances and capabilities  multiple sensor fusion ACC-based gesture control Well suited to distinguish noticeable, larger scale gestures EMG-based gesture control Contain rich information about hand gestures of various size scales We considered the complementary features of ACC (recognition) and EMG (segmentation + recognition)

Hand Gesture Recognition Proposed system Segmentation Feature extraction Recognition with HMM

Hand Gesture Recognition Data Segmentation As hand movement switches from one gesture to another one, the corresponding muscles relax for a while Onset threshold Determine active segments Offset threshold Prevent the fragmentation

Feature Extraction: EMG Hand Gesture Recognition Feature Extraction: EMG Frame-based extraction Every EMG channel is filtered by Hamming window Minimize the signal discontinuities 4×n dimensional feature vectors 3rd order auto-regressive coefficients Mean absolute value Window size = 250ms

Feature Extraction: ACC Hand Gesture Recognition Feature Extraction: ACC Normalize the variations in the scale and speed of gesture Scale the amplitude of the data Extrapolate the active segment to 32 points 3×32 dimensional feature vectors

HMM for Recognition Multi-stream HMMs Weight factors Hand Gesture Recognition HMM for Recognition Multi-stream HMMs Weight factors Recognition result

Virtual Game Control

Experiments: Setup Delsys Myomonitor IV sensor system Five subjects Four-channel EMG + a 3D-ACC Five subjects Utilize LR HMMs with five states Built EMG and ACC HMMs independently (weight = 0.5) Training set is collected by 10 repetitions

Experiments Results

Summary Hand gesture recognition can be utilized in natural interaction between human and computers EMG + ACC to achieve real-time hand gesture recognition  Virtual Rubik’s Cube game