1. Intelligent Learning Systems Design for Self-Defense Education
Manan Mehta*, Bhushan Muthiyan*,
Hyeran Jeon*, Kaikai Liu*, Younghee Park*, Jerry Gao*,
Gong Chen‡, Jim Kao‡
*Department of Computer Engineering
‡ Department of Kinesiology
San Jose State University

2. Increasing Security Concerns
Increasing unarmed or armed attacks
Even college campuses are targets for mass shootings
Run, Hide, and Fight by the Department of Homeland Security
Individuals should know how to protect themselves
Self-defense courses in colleges
Martial arts studios

3. Limitations of Existing Solutions
Lab with 360 degree camera doesn’t fit well with self-defense education
Can capture movement of only single student at a time
Expensive; can’t use the facility outside the lab
Hidden area problem; depending on camera angle
Slow; compute-intensive image recognition for the movement modeling

4. Related works in Vision based approach
Inferring the gestures through image segmentation of hand or body joints
ICCV13: Real-time articulated hand pose estimation using semi-supervised transductive regression forests
CVPR14: Latent regression forest: Structured estimation of 3d articulated hand posture
ACM Trans on Graphics 2014: Real-time continuous pose recovery of human hands using convolutional networks
ACM Trans (TIST) 2014: A Real-Time Hand Posture Recognition System Using Deep Neural Networks

5. Related works in Motion based approach
Wearable products: Jawbone UP3 and UP4, FitBit, Apple Watch, Microsoft Band, and GoQii.
TuringSense: tennis players
Pears: great workouts by the family of coaches
mThrow by Motus measures up to 100 metrics points of stress on joints for baseball players
Sensoria Smart Socks woven three soft pressure sensors and a magnetic Bluetooth electronic anklet snaps
Moov now straps on your wrist to help you analyze running, swimming, cycling and sleeping by providing a unified App.
most of them are focusing on single player motion but we are targeting two-body based motion analysis

6. Related works in Motion based approach
IMU based approach
IEEE Sensors 12: MEMS accelerometer based nonspecific-user hand gesture recognition
IEEE Sensors 14: 2D human gesture tracking and recognition by the fusion of MEMS inertial and vision sensors
ACM MobiSys 14: Risq: Recognizing smoking gestures with inertial sensors on a wristband
most of them are single-node based but we are using multi-node based

7. Proposed Architecture

8. Wearable System Design

9. Major Components
Wearable Device: The designed device contains multiple distributed pea-sized 9-axis inertial measurement units (IMUs) to capture the full motion of forearm, elbow and shoulder.
Smartphone App: The smartphone App collects multi-sensor data, eliminates outliers, and feeds the pre-processed sensor data to the estimation engine, where the coordinated movement of the body’s segments are derived. The derived movement data will be fed into the machine learning pipeline to accurately analyze learning effectiveness and sessions in real time, and deliver real-time metrics as the feedback for the trainers.
Cloud Dashboard: The Dashboard in the web shows summaries of the training data per- session. The collected training data will be compared with the demo model for the effectiveness and performance evaluation. The dashboard can automatically build the evaluation rules and models just by learning the sensed data from the coach’s demo – combines good and common mistakes- without bringing in IT professionals to write computer code and hand pick gesture features.

10. Preliminary Results

11. Thank you