Feature Extraction Spring Semester, 2010

Accelerometer Based Gestural Control of Browser Applications M. Kauppila et al., In Proc. of Int. Workshop on Real Field Identification, UCS 2007, pp. 2-17, 2007.

Outline 4 Motivation 4 Previous Work 4 H/W Architecture 4 Recognizer 4 Experiments 4 Discussion

Motivation 4 Large screen interface –Train, bus stations, marketing places, and other public places –Provide different kinds of interactions between the users and the displays

Previous Work 4 DBN, SVM –Samsung AIT, “Two-stage Recognition of Raw Acceleration Signals for 3-D Gesture- Understanding Cell Phones,” Classify {0~9, O, X} by using DBN SVM is used for the confusing pair of (6, O)

Communication Architecture 4 Scenario 4 Data flow

Communication Architecture 4 Accelerometers –Developed at our university –Send data at 50Hz 4 UPnP virtual sensor 4 Use a simple asynchronous Java API: decouple the recognizer and its clients

Browser Control 4 General browser / photo album app.

Recognizer 4 Segmentation 4 Preprocessing and normalization 4 Classification

Segmentation 4 Feature vector for segmentation Continuous acceleration signal: Discretized acceleration signal: Approximated derivative: Approximated velocity:  4 Two-state (non-gestural / gestural) HMM Recognizer

Segmentation Example Recognizer

Preprocessing 4 Sensor model –Dynamic component (gesture): a d (t) –Static component (gravity): a s =(0, 0, g) T –Measured acceleration: R: orthogonal matrix (Describing the orientation of the sensor) 4 Gravity estimation –Ra s : Mean of the measured acceleration Recognizer

Preprocessing 4 Tilt Compensation –Let u = v/|v|, where v is the axis of rotation –Remember the measured acceleration –Finally, Recognizer

Normalization 4 Power normalization –Frobenius normalization 4 Tempo normalization –Rescale the gesture tempo so that all gestures have 30 samples Downscaling: Box filtering Upscaling: linear interpolation Recognizer

Classification 4 Training set –A single person, 16 samples per gesture 4 Recognizer –12-state hidden Markov model per gesture –Choose the gesture class corresponding to the HMM with the highest score Recognizer

Experiments 4 11 Subjects 4 Confusion matrix

User Study 4 Three stages + a questionnaire (feedback) –Blind stage Freely use the system without any prior training –Task stage Solve a specific browsing task after training –Photo album stage Experiments

Subject Feedback 4 Five questions 4 Free-worded feedback –Stress of hands –Unintuitivity and learning overhead Experiments

Discussion 4 False positives still pose a problem –Rejection mechanism is needed 4 Recoiling problem –Avoiding the use of overly simplistic gestures

Human Activity Recognition with User-Free Accelerometers in the Sensor Networks S. Wang, et al., Int. Conf. Neural Networks and Brain, pp , 2005.

Outline 4 Motivation 4 Feature Extraction 4 Classification 4 Experiments 4 Summary

Motivation 4 Human’s activities can be represented from three aspects –Movements of human bodies –Movements of the objects associated with the activities –Person-object interaction 4 Wearing the sensors is uncomfortable for users

Feature Extraction 4 Using a sliding window with 50% overlap 4 19 features –Six features from each of the three axes Acceleration, mean, standard deviation (stability), energy (data periodicity), frequency-domain entropy, correlation  normalized into [-1.1] –One feature represents vibration of the sensor (|a x 2 +a y 2 +a z 2 -g 2 |)

Classification 4 Recognition algorithms –C4.5, MLP, SVM 4 Three types of tests –Self-consistency test: Training set = test set –Cross-validated test –Leave-one-subject-out validation

Experiments 4 System setup –Accelerometer: KXP74 (32Hz, -2g~+2g) –Fixed to the rear of the telephone receiver, base of the cup, and on the top of the pen 4 SVM-based feature selection

Data collection 4 Three activities was performed by four subjects –Drinking, phoning, and writing –Positive actions + negative actions Positive: Write on the table or on the blackboard Negative: Rotate the pen with fingers, … –Each lasted 5 minutes Experiments

Results 4 Self-consistency test: Accuracies > 95% 4 Cross-validated test 4 Leave-one-subject-out validation test Experiments

Discussion on Feature Selection 4 Attribute sequence –Drinking –Phoning –Writing A: Acceleration, E: Mean, S: Stdev., G: Energy, P: Entropy, C: Correlation, Delta: Vibration Experiments

Summary 4 Accelerometer based gestural control of browser applications –Segmentation feature Acceleration, derivative, and velocity –Tilt compensation 4 Human activity recognition with user-free accelerometers in the sensor networks –Feature extraction and selection