Intelligent Learning Systems Design for Self-Defense Education Manan Mehta*, Bhushan Muthiyan*, Hyeran Jeon*, Kaikai Liu*, Younghee Park*, Jerry Gao*, Gong Chen‡, Jim Kao‡ *Department of Computer Engineering ‡ Department of Kinesiology San Jose State University

Increasing Security Concerns Increasing unarmed or armed attacks Even college campuses are targets for mass shootings Run, Hide, and Fight by the Department of Homeland Security Individuals should know how to protect themselves Self-defense courses in colleges Martial arts studios

Limitations of Existing Solutions Lab with 360 degree camera doesn’t fit well with self-defense education Can capture movement of only single student at a time Expensive; can’t use the facility outside the lab Hidden area problem; depending on camera angle Slow; compute-intensive image recognition for the movement modeling

Related works in Vision based approach Inferring the gestures through image segmentation of hand or body joints ICCV13: Real-time articulated hand pose estimation using semi-supervised transductive regression forests CVPR14: Latent regression forest: Structured estimation of 3d articulated hand posture ACM Trans on Graphics 2014: Real-time continuous pose recovery of human hands using convolutional networks ACM Trans (TIST) 2014: A Real-Time Hand Posture Recognition System Using Deep Neural Networks

Related works in Motion based approach Wearable products: Jawbone UP3 and UP4, FitBit, Apple Watch, Microsoft Band, and GoQii. TuringSense: tennis players Pears: great workouts by the family of coaches mThrow by Motus measures up to 100 metrics points of stress on joints for baseball players Sensoria Smart Socks woven three soft pressure sensors and a magnetic Bluetooth electronic anklet snaps Moov now straps on your wrist to help you analyze running, swimming, cycling and sleeping by providing a unified App. most of them are focusing on single player motion but we are targeting two-body based motion analysis

Related works in Motion based approach IMU based approach IEEE Sensors 12: MEMS accelerometer based nonspecific-user hand gesture recognition IEEE Sensors 14: 2D human gesture tracking and recognition by the fusion of MEMS inertial and vision sensors ACM MobiSys 14: Risq: Recognizing smoking gestures with inertial sensors on a wristband most of them are single-node based but we are using multi-node based

Proposed Architecture

Wearable System Design

Major Components Wearable Device: The designed device contains multiple distributed pea-sized 9-axis inertial measurement units (IMUs) to capture the full motion of forearm, elbow and shoulder. Smartphone App: The smartphone App collects multi-sensor data, eliminates outliers, and feeds the pre-processed sensor data to the estimation engine, where the coordinated movement of the body’s segments are derived. The derived movement data will be fed into the machine learning pipeline to accurately analyze learning effectiveness and sessions in real time, and deliver real-time metrics as the feedback for the trainers. Cloud Dashboard: The Dashboard in the web shows summaries of the training data per- session. The collected training data will be compared with the demo model for the effectiveness and performance evaluation. The dashboard can automatically build the evaluation rules and models just by learning the sensed data from the coach’s demo – combines good and common mistakes- without bringing in IT professionals to write computer code and hand pick gesture features.

Preliminary Results

Thank you