Bio Com Project Sarath Chandra Committee: Dr. Krishna Kavi Dr. Robert Akl Dr. Yuan Xiaohui

Objective  Objective of my thesis work is to develop a sensor system that allows soldiers to communicate and convey information through signals and gestures when there is no direct Line of Sight.  The signals and gestures can be transmitted wirelessly to a HUD(heads up display) which is mounted on the soldiers head.

Outline  Motivation  Gesture set  Standardized hand signals  Components  Previous Works  Analysis & Classification  Summary  Future Work

Motivation  Tactical Squads and SWAT teams involve in high-risk situations such as Hostage rescues, VIP protection, counter terrorist situations etc.  Verbal communication is not feasible in such high-risk situations because:  Absence of Line of Sight  Interference of sounds like gun firing  Teams divide into different groups

Gestures  Gestures are a form of non-verbal communication in which visible bodily actions are used to communicate important messages.  Gesture classification has many important uses in our day to day lives like:  Human computer interaction(HCI) applications  Controlling of prosthetic arms and limbs  Gaming purposes like Wii

Gesture Set  Used the standardized hand signals for close range engagement operations used by military and swat teams.  Gesture set is broadly divided into finger and arm movements.

Standardized Hand Signals

Sensors The different core components of our prototype are:  Surface EMG sensors  Bend(flex) Sensors  Accelerometers

Surface EMG sensors Surface EMG is electrical activity produced by muscle activity and measured by sensors directly attached to the skin

Electromyography  Electromyography(EMG) measures the muscle electrical activity during muscle contractions as electric potential between ground and sensor electrodes.  Electrical activity is typically in the range of 0-90 micro volts range and contained within 0-200Hz spectrum.  2 types of EMG: (a) Surface EMG (b) Invasive EMG

Bend Sensors These devices use a piezo-resistive method of detecting the bend. The sensor output is determined by both the angle between the ends of the sensor as well as the flex radius.

Accelerometers Senses dynamic acceleration(or deceleration) and inclination(tilt) i.e. acceleration due to gravity ) in 3 dimensions simultaneously

Communication  Wi-micro dig enables us to interface sensors wirelessly to our computer.  Designed to obtain data from sensors for measurement and analysis.  It translates analog sensor signals with high resolution into digitally encoded music industry compliant MIDI messages.  The wireless communication protocol followed is Bluetooth.

Wi-micro Dig  A thumb sized hardware device that encodes analog voltage signals generated by up to 8 sensors.

Data Format for MIDI Protocol Infusion system’s I-Cube X Wi-micro Dig MIDI implementation uses the following data format for all system exclusive messages: Byte Description 240(F0h) System exclusive status 125(7Dh) Manufacturer ID {DEV} Device ID {CMD} Command or Message ID [BODY] Main Data 247(F7h) End of System exclusive

Other current research works  Most of the current research going on using EMG for gesture classification is done for various applications like HCI, gaming, controlling of prosthetic arms and limbs based on the electrical information stored in the nervous system etc.  One of the notable work was done by Microsoft in which they use a wearable electromyography based controller that senses the electrical activity produced in muscles using surface EMG sensors and these signals provide a muscle computer interface for interaction with one or more computing devices.

Microsoft Patent  A wearable electromyography based controller is used for interacting with one or computing devices.  The wearable electromyography based controller directly senses and decodes electrical signals produced by human muscular activity using surface EMG sensors.  The controller consists of surface EMG sensor nodes integrated into a wearable band like an armband or a wrist watch.

Wearable EMG based controller

How our prototype works  EMG sensors are placed on the upper and lower forearm. These sensors assist in the classification of wrist movements.  Bend sensors are attached to a glove that is worn on the hand. Sensors are located over the first knuckle of each finger and determine weather a finger is partially bent partially, completely or the finger is straight.  Accelerometer is attached to the glove at the base of the wrist. By converting the signal into a 3 –element vector and using a 3D Mapping technique, it can determine the position of the speed and direction of the arm movements.

ANALYSIS and CLASSIFICATION

 The sensor data is ported into the computer and analyzed with MATLAB. A custom GUI is built to control the overall system.

Analysis Methods Classification is accomplished in Mat lab using a variety of methods:  Fourier Analysis  Threshold analysis  Support vector machines  Naïve Bayes  Wavelet Transforms

Phase 1  Develop a prototype using off the shelf components from I-cube X that can measure EMG as a result of hand movement and convert it into an electrical signal that can be transmitted wirelessly.

Two channel EMG sensor system plot

Phase 1 cont … Some of the lessons learnt from phase one are:  Succeeded in developing an interface to receive EMG signals and analyze using MATLAB  Crosstalk between muscles impaired localization  Vey low classification accuracy  Signal attributes found are: Raw EMG,FFT

Phase 2  Investigate the current state of the art EMG sensors and an in depth analysis of different feasible implementations and solutions to this project.

Raw EMG,FFT, Cepstrum

Phase 2 cont… Some of the lessons learnt from phase two are:  Clinical EMG sensors were found to be much more accurate  Understanding gained from these sensors as well as further research allowed us to modify and reconfigure the EMG sensor system.  Increased classification accuracy  Classification techniques were identified like: Logistic regression, SVM, Neural networks

Phase 3  Develop a prototype with either new more accurate sensors or the optimized Bio flex sensors.  Develop a classification technique to improve classification.

3 channel EMG system

Phase 3 cont.. Some of the lessons learnt from phase 3 are:  Using optimized bio flex sensors a new prototype was created using 3-5 sensors  Using SVMs and other techniques several revolutions of increased accuracy  Limiting the number of features to abstract and percentage based components FFT component

Phase 4  Constructed a 5 channel EMG sensor prototype  Took more samples involving three subjects for 5 gestures.  Primary and the secondary sensors were selected.  Mean, variance and standard deviation were calculated for each gesture of the respective samples.  Found very high variance and standard deviation involving samples of all the 3 subjects taken at once.

Phase 4  Wavelet transforms are used to find the difference between these gestures.  Signal features present in the detail and approximation coefficients of the respective signals for each gesture are similar with slightly different features for each gesture.  Flex sensors are attached to a glove that is worn on the hand.  Located on the first knuckle of every finger and can determine weather a finger is bent partially, completely or when there is no bend.

Gesture 1

Gesture 3

Phase 4 cont …  Took the difference between samples of different gesture pairs to find notable difference between the gestures.  Plotting of the difference between samples showed that each gesture has noticeable amplitude spikes in a very narrow band width of Hz range.  These narrow bandwidths are so congested that it becomes difficult to get higher accuracies above 75%.

Difference between Gesture 3 & 1

Phase 5  Wavelet transforms were used to find any more noticeable differences between the gestures.  The approximation and detail coefficients of these gestures are almost similar with slightly different waveforms.  Incorporated additional sensor types such as flex sensors and accelerometers.

Phase 5  Thresholds are defined to determine the bent of the finger.  95% classification accuracy achieved.

Summary and Conclusion  Electromyography can be used effectively for the classification of wrist movements with high degree of accuracy.  Flex Sensors can be used to classify the finger movements effectively because they clearly distinguish the fingers which are bent versus the fingers which are not bent thus classifying the finger gestures effectively.

Future Work  Use a fully integrated sensor system including the accelerometers to classify the complete gesture set with high degree of accuracy.  Transmit the messages wirelessly and securely to display it on a HUD  Design an encryption scheme for secure transmission to the HUD.

Questions??