Toward Brain-Computer Interfacing Minhye Chang

Contents 5. The Berilin Brain-Computer Interface: Machine Learning-Based Detection of User Specific Brain States 7. Brain Interface Design for Asynchronous Control

Chapter 5 Berlin Brain-Computer Interface procject Premovement Potentials in Executed and Phantom Movements BCI Control-Based on Imagined Movements Lines of Further Improvement

Brelin Brain-Computer Interface

EEG-based BCI system with machine learning techniques – High quality feedback w/o subject training – Copes with the huge intersubject variability Spatial resolution of the somatotopy Discriminability of premovement potentials in voluntary movements Sensorimotor rhythms caused by motor imagery – 128 channel EEG – Subjects w/ no or very little experience w/ BCI control – Information transfer rate above 35bits per minute.

Premovement Potentials Bereitschaftspotenzial, RP (readiness potential) – Activity in the motor cortex leading up to voluntary muscle movement – To build a classifier Letting healthy subjects actually perform the movements – Movement imagination poses a dual task: motor command preparation plus vetoing the actual movement Predictions about imminent movements – Exclude a possible confound with feedback from muscle and joint receptors – Assistance of action control in time-critical behavioral contexts

Left vs. Right Hand Finger Movements Readiness potential – Pronounced cortical negativation Left-hand vs. Right-hand finger tapping experiment – Electrodes : CCP3 and CCP4 – Predominantly contralateral negativation before movement

Self-paced finger-monements on a computer keyboard – Tap-rates of 30, 45, 60, and 120 taps per minute – 128 Ag/AgCl scalp electrodes – EMG, EOG Discriminate premovement potentials as fast as two taps per second Highly subject-specific Left vs. Right Hand Finger Movements

Preprocessing Extracts the low-frequency content w/ an emphasis on the late part of the signal

Classification Regularized linear discriminant analysis (RLDA) – RP features normally distributed with equal covariance matrices. – The data processing preserve gaussianity

RP-Based Feedback in Asynchronous Mode Calibration session – The data is used to train a classifier. A useful continuous feedback in an asynchronous mode – Classifier must work for a broader interval of time – The system needs to detect the buildup of movement intentions Quite simple strategy : jittering – Extracts several with some time jitter b/w training samples – Two samples per key press: at 150 and at 50 ms before key press – Invariant to time shifts EMG activity at about 120ms

RP-Based Feedback in Asynchronous Mode Movement intention detector – Distinguishes b/w motor preparation intervals and “rest” intervals In a BCI feedback experiment (-160 to -80ms)

Detection of ‘Phantom Limb Commands’ ERD(ERS) Attenuation (amplification) of pericentral μ and β rhythms in the corresponding motor areas Lack of a time marker signal – Listened to an electronic metronome – Deep sound: rest, higher sound: perform either a finger tap or a phantom movement 8 patients with amputations showed significant ″ phantom- related ″ ERD/ERS of μ- and/or β -frequencies – Signed r 2 values of the differences in ERD curves

Gross somatotopic arrangement – Hand versus foot Finely graded representation of individual fingers Examine the discriminability of BCI signals from close-by brain regions – 128-channel EEGs – During self-paced movements of various limbs – Significantly reflect specific activations in sensorimotor cortices Exploring the Limits

Averaged premovement potential patterns of one subject in different self-paced limb moving tasks Significantly reflect specific activations in sensorimotor cortices

In repetitive movements, the discrimination decays already after about 1s Modulations of sensorimotor rhythms evoked by imagined movements 6 subjects who had no or very little experience w/ BCI feedback – 118-channel EEGs – Recorded EOG and EMG Calibration measurement (machine training) – Estimate parameters of a brain-signal to control-signal translation algorithm BCI Control-Based on Imagined Movements

In the training sessions – Visual stimuli for 3.5s: (L) left hand, (R) right hand, or (F) right foot – Two types were selected for feedback; binary classifier Bias and scaling of the linear classifier – Different experimental condition of the (exciting) feedback situation Experimental Setup

1 st feedback application: position-controlled cursor – Classifier output translated to the horizontal position of a cursor. Experimental Setup Holding for 500ms ActivatedSuccess

2 nd feedback application: rate-controlled cursor – At each update step a fraction of the classifier output was added to the actual cursor position. 3 rd feedback application: basket game – Operated in a synchronous mode – Horizontal position was controlled by the classifier output Experimental Setup

Spatial filters – Optimize the discriminability of brain signals based on ERD/ERS effects of the motor rhythms – Common spatial pattern (CSP) analysis Feature calculation – Log of the variance in those surrogate channels – Calculated every 40 ms from sliding windows of 250 to 1000ms (subject-specific) for online operation Details about the processing methods and the selection of parameters : Blankertz et al. (2005) Processing and Classification

Information transfer rate (ITR) in bits per minute (bpm) – Compared to ROC curves, ITR considers different duration of trials and different number of classes Highest ITRs: rate-controlled cursor, asynchronous protocol Results

It can be operated at a high decision speed – Average trial length for 1 st, 2 nd, and 3 rd was 3s, 2.5s, and 2.1~3s resp. The fastest subject : average speed of one decision every 1.7s Subject who showed the most reliable performance – only 2% of the total 200trials were misclassified at an average speed of one decision per 2.1s – Sentences 135 characters in 30 minutes (4.5 letters per minute) in a free-spelling mode Results

It is possible to voluntarily modulate motorsensory rhythms w/o concurrent EMG activity (Vaughan et al., 1998) Squared biserial correlation coefficient, γ 2 – For the classifier output and for the bandpass filtered and rectified EMG signals of the feedback sessions – Occurrence of minimal EMG activity in some trials does not correlate with the EEG-based classifier Investigating the Dependency

CSP with Simultaneous Spectral Optimization – CSP strongly depends on the choice of the bandpass filter – Broadband filter for general choice – Subject-specific choices – Optimized spatial filters (usual CSP technique) + temporal finite impulse response(FIR) filter : enhance the discriminability Significant superiority of the proposed CSSSP The spatial and/or the spectral filter can be used for source localization of the respective brain rhythms. CSSSP

Nonstationarities in EEG signals – How much of this nonstatioarity is reflected in the EEG features – How strongly is the classifier output affected – How can this be remedied The most serious shift occurred b/w the initial calibration measurement and online operation Shifts during online operation were largely compensated for by the CSP filters or the final classifier Simple adaption of classification bias successfully cured the problem. Need for Adaptivity

Chapter 7 Introduction Asynchronous Control EEG-Based Asynchronous Brain-Switches Asynchronous Control Design Issues

Neil Squire Society – The only national not-for-profit organization in Canada – Neil Squire Brain Interface lab focused on BI system design specifically for asynchronous control environments. Synchronized control environments where the system dictates the control of the user Robust multistate, asynchronous brain-controlled switch in the most natural manner in the real-world environments Introduction

Asynchronous Control

Output signal levels are changed or commands are issued only when control is intended – Intentional control (IC) / no control (NC) state – Remains neutral or unchanged during the NC state Asynchronous control – Characteristic of most real-world control applications – Most people expect from interface technology Asynchronous Control NC state; stable and unchangedIC; available for control

The neutral or unchanging system output response during NC states Poor idling is indicated by false switch activations How well BI transducers idle – Rate of false activations or false positive error True and false activation rates as performance metrics System Idling With no gas (the NC state), the engine idles. Poor at idling

Synchronized BI system – System-driven control strategy which cause significant user frustration and fatigue Asynchronous TV controller – Simply change channels at any time they wish – The channel selection is stable when users are watching TV (NC state) Synchronous TV controller – Regularly poll the user to ask – Renders the changing of channels to specific periods Synchronized BI System

Two serious drawbacks – control decision regardless of whether the person is actually intending control – User would need to wait for the system polling period to occur Synchronized BI System

Essence of the temporal control paradigms – Idle support and system availability Continuously available control – Always ready for the user to control Periodically available control – For Initial trial-based technology development – For restricting the signal processing complexity – Blocks a user’s attempt to control for the periods b/w control periods – “no control is possible at this time” System availability

Unintended action during NC states – Control paradigms that do not support idling – “Midas touch problem” by the eye-tracking community Four primary control paradigms – Constantly engaged mode : impractical – Asynchronous mode : most natural assistive device operation Control Paradigms

Specific signal processing algorithms to handle the NC state – Optimized TP rate and minimized FP rates – Trade-off b/w TP accuracy and FP rates → specific characteristics of an application Brain-switch based on the outlier processing method(OPM) – Extract single-trial voluntary movement-related potentials (VMRPs) – TP rates greater than 90% – FP rates b/w 10 and 30% → limits OPM as an asynchronous switch – Multiposition asynchronous brain-switch Asynchronous Brain-Switches

Attempted VMRPs – Voluntary movement : existing and natural internal control system – “attempted” : subject with SCI attempt to move their fingers – By a very different neural mechanism The low-frequency asynchronous switch design (LF-ASD) – Relative power in the 1-4Hz band from ensemble VMRP increased – Focus on low frequency band – Wavelet analysis of the EEG signal over SMA and MI – Lower end of FP activations – TP rates of 30 to 78% during IC states with very low FP rates of 0.5 to 2% during NC Asynchronous Brain-Switches

Additional signal processing – EEG signal normalization – Switch output debounce – Feature set dimensionality reduction blocks – Increased the TP rate by an average of 33% Feature vectors navigate the feature space – System classification accuracy of more than 97% Levine and Huggins, Millán et al. (2004a), Yom-Tov and Inbar (2003), and Townsend et al. (2004) Additional signal processing

Very low FP rates during the NC state – High FP error rates cause undue user frustration – False activation appear uncontrollable to the user – Subjects would rather experience more trouble performing accurate hits ( low TP rate) Evaluating asynchronous control – Receiver operating characteristic (ROC) curves – BI transducers with low FP error rates Asynchronous Control Design Issue1

“Tuning” the performance – ROC curve shows possible operating setups – by tuning various parameters Two-state LF-ASD brain-switch (on/off) – Scaling the relative magnitudes of NC state feature vectors vs. IC state ones – FP rates under 1% during the time b/w FPs (30s or more) Asynchronous BI Performance

FPs typically clump together – Large periods of system idle time with free of FPs Switch-output jitter reduction methods – Switch debounce block – Improve error rates by reducing the FP jitter in the switch output – Trade-off : transducer availability – Debounce time ↑ → time the transducer is available ↓, control time ↓ Switch-output jitter reduction Borisoff et al. (2004)

Intra-False Positive Rates – Dependent on the output classification rate of a transducer – FP rate of 1% with classification output every 1/16 th sec → FP every 1/16 * 100 = 6.3s Average time b/w errors greater than 30s – Reasonable design goal : FP rate of under 0.25% False activation rates : time rates w/ raw percentages Asynchronous Control Design Issue2

Temporal characteristics of NC and IC states – Environmental controllers : periods b/w the IC commands – Neural control of a robotic device : intercontrol times during periods of intense usage Ubiquitous ON/OFF problem – Users with amputation have to turn the system on/off by themselves – Confirms user intent to turn the system to the awake mode – Eliminate FPs and require a simple commands Asynchronous Control Design Issue3

Costs associated with an FP – Multiple operating levels of the sleep mode – Full awake mode : sequence through higher modes, higher FP rates and intentional command sequences How to structure tasks during different phases – Customization, training, and testing phases Asynchronous Control Design Issue4 Sleep mode Awake mode Low costs of FPs Easily corrected High FP rates Less intricate

Initial customization – Accurate time-stamping for calibration and classifier – System-guided and system-paced tasks – Train and test in very similar control environments Apparatus necessary – Self guided and self-paced tasks → Self-report errors – Contamination of data – Accurate assessment of asynchronous system performance Asynchronous Control Design Issue4

Thank you for your attention