Department of Computer Science San Diego State University Real-Time Feature Extraction and Classification of Prehensile EMG Signals Master Thesis Christopher Miller Supervisor: Marko Vuskovic Department of Computer Science San Diego State University March 24, 2008 Good morning, I’m Christopher Miller and today I will present the defense of my thesis: Real-Time Feature Extraction and Classification of Prehensile EMG Signals.

Copyright © 2008 by Christopher Miller Agenda Introduction Electromyography (EMG) Signals EMG Signal Processing Classification Experimental Results Implementation Conclusion Future Work Questions Here is the agenda that I will follow. Following a brief introduction, I will explain Electromyography or EMG signals and how they are acquired I will then cover various approaches to EMG signal processing and classification, and explain in detail the methods employed in this research I will show you the results I achieved through parameter optimization with offline testing as well as the results of online testing using the program developed for this thesis I will explain the features of that program and show a short video to demonstrate its effectiveness Finally, I will conclude and offer suggestions for future research and then answer any questions that remain. Copyright © 2008 by Christopher Miller

Copyright © 2008 by Christopher Miller Introduction (1 of 2) Numerous technological advances in prosthetic hands Greater degrees of freedom Continue to function as pincers There have been numerous technological advances with prosthetic and robotic hands. On the left is depicted the Dorrance hook hand: a 1 degree of freedom pincer. In the middle is the 6 degree of freedom SDSU hand On the right is the best example of the advances made in prosthetics: the Touch Bionics I-LIMB hand, with individually powered fingers. Unfortunately, while the powered fingers enable more advanced grasps, the device is controlled by only 2 electrodes, which detect grasp opening and closing. It essentially is still a pincer and places unnecessary mental and physical demands on the operator. Copyright © 2008 by Christopher Miller

Copyright © 2008 by Christopher Miller Introduction (2 of 2) The purpose of this thesis was to implement a program that could perform real-time feature extraction and classification of prehensile EMG signals for the following grasps: With the addition of 2-3 surface electrodes, a control system could preshape the prosthetic hand while the operator approaches the target object. The program recognizes four of Schlesinger’s grasp classifications: spherical, cylindrical, precision or disk, and lateral or key To do this, it analyzes EMG signals from the operator’s forearm. Copyright © 2008 by Christopher Miller

Electromyography (EMG) Signals: Myoelectric Energy Detection Motor units control groups of muscle fibers Brain recruits motor units to innervate muscles for movement Myoelectric energy produced as motor units activate Surface electrodes detect myoelectric energy The motor unit is the fundamental building block of the EMG signal. It consists of a motor neuron, its axon, and a number of muscle fibers that it innervates. The muscle fibers are intermingled with the muscle fibers of other motor units. When the brain sends signals to contract a muscle, an increasing number of motor units are recruited to contract their muscle fibers, proportional to the speed and tension required. The resulting exchange of ions across the innervated muscle fibers produces a small electrical current, known as the motor unit action potential or MUAP. The sum of all MUAPs is referred to as myoelectric energy, a recording of which is an electromyograph. We obtain this EMG signal through the use of surface electrodes. Copyright © 2008 by Christopher Miller

Electromyography (EMG) Signals: EMG Amplification SENIAM Recommends Pre-gelled Ag/Ag-Cl Bipolar 0.8” inter-electrode distance 0.4” wide Surface electrodes 1” inter-electrode distance 0.875” wide EMG amplification device (Saksit Siriprayoonsak, 2005) 4 bipolar channels 1 reference channel SENIAM: Surface EMG for Non-invasive Assessment of Muscles Produced a number of recommendations for EMG collection in 1999. The electrodes used for this research do not meet their specifications, but still perform fairly well. The EMG amplifier, depicted on the bottom right, was built in 2005 by Saksit Siriprayoonsak. It reads up to 4 bipolar channels and uses 1 reference channel for noise cancellation In order to capture the correct signals, one has to have a basic understanding of the muscle locations Copyright © 2008 by Christopher Miller

Electromyography (EMG) Signals: Muscle Anatomy These are the muscles that control the shaping of a grasp: Extensor digitorum – extends all four fingers, primarily middle and ring, plus the wrist Extensor pollicis longus and brevis – control thumb extension and abduction at wrist Extensor indicis – extension and adduction of the index finger Extensor digiti minimi – extension of little finger Note that the Extensor indicis is in a deeper muscle layer than extensor digiti minimi and extensor digitorum, making it more difficult to detect Copyright © 2008 by Christopher Miller

Electromyography (EMG) Signals: Mounted EMG Amplifier Here is what the mounted amplifier looks like with one possible electrode arrangement The final arrangement for an individual should be determined through testing of potential electrode sites while moving the corresponding fingers. We are now ready to collect and analyze the EMG data Copyright © 2008 by Christopher Miller

EMG Signal Processing: System Flowchart The EMG signal is in analog form and is converted to digital form using an A to D converter card, which in our case, is a National Instruments 6220 multi-function DAQ card. The analog signal is sampled at 1 KHz. It then undergoes signal processing, which involves time sample extraction, feature extraction, and transformation, before classification is performed. The prosthetic or robotic hand controller will be created in future research and act on the classifier’s determination. Copyright © 2008 by Christopher Miller

EMG Signal Processing: Onset of Movement Detection Lidierth (1986) Hodges & Bui (1996) Bonato et al. (1998) Staude et al. (2001) – AGLRamp, AGLRstep In our case, time sample extraction is really just detecting the onset of movement and extracting the following 400 ms. There are several approaches here, all of which were investigated by Staude. His research showed that his statistical approaches were optimal, but the Bonato method performed almost as well and is much simpler to implement. Copyright © 2008 by Christopher Miller

EMG Signal Processing: Bonato Method The Bonato method adds the square of each even sample to the square of the following odd sample and divides by the rest position variance. Given an m length segment, if the threshold, h, is surpassed for at least n times, then the segment is considered active. If the signal remains active for T1 samples, then the onset of movement has been detected and the first sample over h of the first active segment is designated as the onset point With the 400 ms of EMG data, we can now extract salient features for classification Copyright © 2008 by Christopher Miller

EMG Signal Processing: Feature Extraction Methods Mean Absolute Value (Hudgins et al., 1993) Mean Absolute Value Slope (Hudgins et al., 1993) Willison Amplitude (Willison, 1964) Zero Crossings (Hudgins et al., 1993) Slope Sign Changes (Farry et al., 1996) Waveform Length (Farry et al., 1996) Simple Square-Integral Amplitude of the First Burst (Vuskovic et al., 2002) Multiple Time Windows (Du et al., 2003) Short-Time Fourier Transform Wavelet Transform Wavelet Packet Transform Spectral Moments (Vuskovic et al., 2005) There are numerous feature extraction methods. Some are simple temporal or time-based approaches, while others are frequency based, which are generally more computationally intensive. I compared the temporal approaches to the spectral moments method developed by Dr. Vuskovic. Copyright © 2008 by Christopher Miller

EMG Signal Processing: Feature Extraction Comparison Method Parameters 400 ms 300 ms 200 ms MAV 96.11% 95.56% 93.89% MAVSLP I = 3 83.89% 81.67% 73.89% VAR 90.56% 90.00% 86.67% WAMP H = 29 98.33% 92.22% Wave form L ength 98.89% 95.00% Zero - Crossings H = 28 94.44% Slope Sign Changes H = 100 96.67% 86.67% For testing, I used 180 EMG recordings from a single individual, which consisted of 30 recordings for each of six grasps: large sphere, small sphere, large cylinder, small cylinder, disk, and key. Each recording consisted of 4 channels and 400 ms of data. The large and small grasps were not distinguished. I performed leave-one-out validation using a Mahalanobis distance based classifier for each possible feature vector with several different time samples. The Waveform Length feature performed particularly well, even at 300 ms. Spectral moments performed rather poorly in this first test, but has several tunable parameters which warranted further investigation. Both methods are employed in the real-time program. Squared Integral 90.56% 90.00% 86.67% Spectral Moments m = 1 92.78% 89.44% 82.22% K = 11 flag = 0 Copyright © 2008 by Christopher Miller

EMG Signal Processing: Feature Extraction Methods Employed (1 of 3) Spectral Moments: I-coefficients, calculated in advance: The traditional method of calculating a spectral moment requires the use of the power spectral density (PSD) function based on the discrete-time Fourier transform However, Dr. Vuskovic found another approach, which uses the Wiener-Khintchine representation of the PSD. Css is the autocorrelation function. The spectral moments are calculated with this function, which relies on I-coefficients that can be calculated in advance. Vuskovic also suggested a linear transformation to reduce error from noise, called Reduced Moments Copyright © 2008 by Christopher Miller

EMG Signal Processing: Feature Extraction Methods Employed (2 of 3) Spectral Moment Parameters Here are the tunable parameters for spectral moments. Copyright © 2008 by Christopher Miller

EMG Signal Processing: Feature Extraction Methods Employed (3 of 3) Waveform Length: Waveform Length is much easier to calculate. It is the sum of the changes in amplitude for the entire signal. Metaphorically, it is equivalent to grabbing both ends of the waveform like a string and pulling it straight. Copyright © 2008 by Christopher Miller

Classification: Methods Artificial Neural Networks (McCulloch-Pitts, 1943) ARTMAP Networks (Grossberg et al., 1976) Mahalanobis-Distance Based ARTMAP Network (Xu et al., 2003) Maximum Likelihood Estimation (Fisher, 1912) Mahalanobis Distance (Mahalanobis, 1936) There are also several classification methods that have had success in classifying EMG data. Of particular note is the Mahalanobis-distance based ARTMAP network, which is well suited for online learning. However, with greater focus on the feature extraction method chosen, only the Maximum Likelihood Estimation and Mahalanobis distance classifiers were used in this research Copyright © 2008 by Christopher Miller

Classification: Methods Employed Maximum Likelihood Estimation (MLE) Mahalanobis Distance MLE attempts to maximize the likelihood of generating the observed data given a specific set of parameters. The probability density function is calculated here, where Φ (fee) is the maximum likelihood estimator, Sc is the covariance matrix of the training set, and mu is the mean of the training set The Mahalanobis distance is actually a maximum likelihood estimator itself…it is calculated using part of the exponential portion of MLE. Copyright © 2008 by Christopher Miller

Experimental Results: Impact of Log Transformation After optimizing the parameters of spectral moments, the best classification rate I could achieve was 92.78% at 400 ms. However, performing a post feature extraction transformation significantly improved the results. On the left, I’ve plotted the zero moments of the first three channel of EMG data. On the right is the same data following a logarithmic transformation Copyright © 2008 by Christopher Miller

Experimental Results: Box-Cox Transformation In an Analysis of Transformations, Drs Box and Cox proposed an alternate power transformation, based on a lambda parameter. Notice that if lambda is zero, then a log transformation is performed. The top equation is applicable if all of the data is positive, which is the case with spectral moments Copyright © 2008 by Christopher Miller

Experimental Results: λ Optimization Mahalanobis-distance Classifier using Spectral Moments Flag = 0 K = 11 Ts = 250 Comparing log to Box-Cox, we see that for small lambda values, the classifier’s performance is the same. Based on this and other tests, I selected a lambda value of 0.06. No significant distinction from different lambda values at 400 ms, so 250 ms was used Copyright © 2008 by Christopher Miller

Experimental Results: Moment Optimization Classifiers using Spectral Moments Flag = 0 K = 11 Ts = 400 Lam = 0.06 To determine the optimal moment parameter, leave-one-out validation was used with m set from 0 to 4 and the other parameters as shown. The optimal moment value was 2, where both classifiers achieve 99.4% Copyright © 2008 by Christopher Miller

Experimental Results: Time Sample Optimization Mahalanobis-distance Classifier using Spectral Moments M = 2 Flag = 0 K = 11 Lam = 0.06 The time sample used also has a significant impact on the classifier’s ability. Of course, to make the real-time program function seamlessly, the time sample needs to be minimized. As you can see, with the Box-Cox transformed spectral moment feature vector, the classifier achieved 99.4% at 300ms. Copyright © 2008 by Christopher Miller

Experimental Results: Channel Reduction (1 of 2) One other method for improving the classifier’s accuracy and efficiency is dimensionality reduction, which is the process of discarding those features that have the least amount of discriminatory power. In this case, I found that removing an entire channel of EMG data was appropriate. As this chart shows, channel 4 had the least amount of variance around the means of each grasp type, meaning the least discriminatory power. Copyright © 2008 by Christopher Miller

Experimental Results: Channel Reduction (2 of 2) Mahalanobis-distance Classifier using Spectral Moments M = 2 Flag = 0 K = 11 Lam = 0.06 3 channels With only 3 channels of data now, the classifier achieved 99.4% as early as 270 ms, a 30 ms improvement without a loss in accuracy. Copyright © 2008 by Christopher Miller

Experimental Results: Feature Comparison (1 of 2) Waveform Length - - Spectral Moments Comparing spectral moments to waveform length now, we see that spectral moments achieved much better results, actually hitting 100% at 264 ms. It is much less stable than waveform length, though, across successive time samples. Copyright © 2008 by Christopher Miller

Experimental Results: Feature Comparison (2 of 2) Wanting to see how a hybrid feature vector performed, I compared spectral moments, waveform length, and 3 hybrids combining waveform length with different numbers of moments. This time, varying levels of training were used. With an increasing percentage of the 180 total recordings used for testing and a corresponding decrease in training, the spectral moments degraded in performance, while waveform length remained relatively unchanged. The hybrids did well, but not well enough to warrant using them. Instead, I decided to use a hybrid approach Copyright © 2008 by Christopher Miller

Experimental Results: Hybrid Approach With small training sets, waveform length is used as the default feature vector. As the training set increases in size, waveform length is compared to spectral moments and the optimal feature is selected This chart depicts the two features’ performance on data collected through my Real-Time EMG Classifier program A total of 40 of each grasp were recorded Spectral moments surpasses waveform length feature at 28 of each grasp for training Copyright © 2008 by Christopher Miller

Experimental Results: Spectral Moments Best classification rate: 97.5% Optimal time sample: 378 ms These next slides illustrate how the comparison is made. With only 20 of each grasp in the training set, Spectral moments achieved 97.5% at 378 ms Copyright © 2008 by Christopher Miller

Experimental Results: Waveform Length Best classification rate: 95% Optimal time sample: 300 ms Waveform length only achieved 95%, but did so with only 300ms of data, so in this case, it would be selected Copyright © 2008 by Christopher Miller

Experimental Results: Waveform Length Online Testing Cross-validation 95% at 300 ms Online validation Sphere: 22/25 (88%) Cylinder: 25/25 (100%) Precision: 21/25 (84%) Lateral: 17/25 (68%) Total: 85% Online testing was interesting. With the wavelength feature used, leave-one-out cross validation achieved 95%, but online testing resulted in only 85%. You can see that the lateral grasp was the hardest to correctly classify. Copyright © 2008 by Christopher Miller

Experimental Results: Spectral Moments Online Testing Cross-validation 89.4% at 355 ms Online validation Sphere: 24/25 (96%) Cylinder: 25/25 (100%) Precision: 25/25 (100%) Lateral: 19/25 (76%) Total: 93% With 40 of each grasp, spectral moments overcame waveform length as the optimal feature. Cross validation scored it at 89.4%, but online testing showed a 93% correct classification rate. Only one spherical grasp was misidentified as a cylindrical grasp and the other 6 misses were from the lateral grasp. Copyright © 2008 by Christopher Miller

Implementation: Program Files The program was implemented in MATLAB, with a C shared library for reading data from the EMG device Copyright © 2008 by Christopher Miller

Implementation: Main Screen Copyright © 2008 by Christopher Miller

Implementation: Training (1 of 3) Green vertical line indicates onset of movement detected using Bonato method. Copyright © 2008 by Christopher Miller

Implementation: Training (2 of 3) Leave-one-out validation from 200-400 ms in 10 ms intervals Optimal time explored at 1 ms intervals Both features scored based on classification rate and time sample required Highest scoring feature selected and system trained Copyright © 2008 by Christopher Miller

Implementation: Training (3 of 3) May load training from files Training recordings automatically stored with classifier Training recordings saved to EMGRecordings folder Copyright © 2008 by Christopher Miller

Implementation: Real-Time Grasp Classification Onset of movement Rest Start Onset of detected Collect Position Grasp movement not grasp data detected Grasp tension Grasp below threshold classified This finite state machine depicts the algorithm for capturing and classifying a single grasp in real-time. Hold Grasp Grasp tension above threshold Copyright © 2008 by Christopher Miller

Implementation: Real-Time Grasp Classification Screenshot Screenshot of single capture Copyright © 2008 by Christopher Miller

Implementation: Real-Time Grasp Classification Video

Copyright © 2008 by Christopher Miller

Copyright © 2008 by Christopher Miller Conclusion Feasible approach demonstrated for real-time classification of prehensile EMG signals More natural approach reduces mental and physical demands on operator Only 3 EMG channels are necessary to classify the 4 grasps Waveform Length proved to be valuable for small training sets Spectral Moments enabled better performance (93% online) with larger training sets and optimized parameters K = 11 m = 2 flag = 0 λ = 0.06 Ts optimized for training set Copyright © 2008 by Christopher Miller

Future Work: Additional Grasps Previous research explored classifying 6 grasps, which included small and large versions of balls and cylinders Two-phased approach to classification is more likely to succeed Copyright © 2008 by Christopher Miller

Future Work: Full Control Channel 4 of current device can be employed for grasp control Copyright © 2008 by Christopher Miller

Future Work: Online Learning Real-Time program should be written in multi-threading capable language Online learning capability with feedback from prosthetic hand Mahalanobis-distance based ARTMAP network suggested Copyright © 2008 by Christopher Miller

Questions

Copyright © 2008 by Christopher Miller

Backup Slides

Differential Amplification Copyright © 2008 by Christopher Miller