Signal processing techniques for fNIRS and application to Brain Computer Interfaces Gautier Durantin, ISAE/CERCO French community for functional NIRS ISAE, Toulouse,

Introduction  EEG and fNIRS today encompass the most active areas of Brain Computer Interfaces research (Min 2010)  fNIRS is currently mainly used as a complement of EEG (Takeuchi 2009, Fazli 2011)  Noise reduction techniques and signal improvement techniques are the next step to improve BCI performance (Mitsukura 2013, Izzetoglu 2010) (Bashashati 2007) 2

Filtering for fNIRS – What do we need to filter ?  Low frequency components : linear trends, measurement bias (Jang 2008)  High frequency components : physiological noise (cardiac frequency), measurement noise (Huppert 2006) FILTERING MODULE Raw signal (x)Filtered signal (y)  Use of filters to remove these components (e.g. linear filters) ! Delay, stability, performance 3

MACD  We propose a specific linear filter used by economists (Utsugi 2007, Cui 2010) Exponential moving average (EMA) Raw signal (x)Filtered signal (y)  The MACD (moving average convergence divergence), based on Exponential moving average (EMA) filters.  MACD is obtained from two EMA : one short (k small), and one long (k big) Short- term EMA Raw signal (x) Filtered signal (y) Long-term EMA +-+- Moving Average Convergence Divergence Filter 4

 Economists use a signal line, obtained from a short EMA (5s) of MACD data, to predict stable increases on the curve. (Appel 1999) MACD (short = 6s ; long = 13s)  A MACD crossover of the signal line predicts a stable increase in the signal measured A MACD crossover of the signal line predicts a stable increase in the signal measured Moving Average Convergence Divergence Filter 5

Real-time hemodynamic response onset detection Stimulus Increase in the signal MACD crossover 9 subjects 24 trials  Controlled experiment of digit sequence memorization task 6 X REST (6-9sec) 1 STIMULATION x-x-x ANSWER (8 sec)

Towards a real-time BCI using fNIRS  Low load (speed 200, heading 200, Alt. 2000…)  High load (speed 245, heading 315, altitude 8600…)  4 control knobs : Speed, Heading, Altitude, Vertical speed 7

Air Traffic Control (simulated) Pilot in ISAE flight simulator fNIRS output Signal filtering and synchronization (MACD) ATC msg m i Resp. windows SOA Load detection TASK or REST Classification Process Data knowledge from phase L Real-time information on pilot’s mental state (Rest VS Task) Towards a real-time BCI using fNIRS 8 Overall accuracy : 98 % (std. Dev : 2,6%)

Air Traffic Control (simulated) Pilot in ISAE flight simulator fNIRS output Signal filtering and synchronization (MACD) ATC msg m i Resp. windows SOA Load detection TASK or REST Classification Process Data knowledge from phase L Towards a real-time BCI using fNIRS 9 Real-time information on perceived workload

Classification process fNIRS output of 20 training trials Signal processing (MACD) and feature extraction  Use of different features (Tai & Chau 2009)  [HbO 2 ], [Hhb], peak response, kurtosis, skewness on different time windows Classifier design (LDA, SVM) Classifier design (LDA, SVM) CLASSIFIER TRAINING TESTING fNIRS output Real-time information on workload 10

Towards a real-time BCI using fNIRS : results 28 sessions 20 training trials 20 testing trials  Overall accuracy obtained during testing phase : 79 % (std. dev : 12,8%)  19 subjects out of 28 have more than 75% accuracy 11

To improve signal processing and BCI accuracy, a solution would be to add a priori information in processing models Further improvements in signal processing  Use of hemodynamic response models for temporal dynamic estimation of fNIRS. (Boynton 1996, Buxton 1997)  Use of Kalman filtering to include estimation of temporal dynamics in signal processing (Abdelnour 2009, Gagnon 2011) 12

Kalman filtering KALMAN FILTER fNIRS raw data fNIRS filtered signal Dynamical model of hemodynamic response and fNIRS measurement Stimulus Physiological processing model (HRF) Participant Measurement model NIRS NIRS signal Confidence in the measures Confidence in the model Kalman filtering 13

 Tested offline on digit span memorization task data (9 subjects) with three levels of difficulty Kalman filtering : results LINEAR FILTERING (effect size eta²=0,2) KALMAN FILTERING (effect size eta²=0,34) Kalman filtering is a promising tool to improve signal useability Challenges remain concerning Kalman tuning and real-time implementation Kalman filtering is a promising tool to improve signal useability Challenges remain concerning Kalman tuning and real-time implementation 14

 Signal processing is a key step towards efficient Brain Computer Interface using fNIRS.  Linear filtering brings good results, but improvements can be made to improve the accuracy of BCI designed with this type of filters.  Kalman filtering or adaptive filtering are the best opportunities to improve signal useability. Conclusion 15

Thank you for your attention 16

17 Digit span task  6 levels of difficulty  4 trials for each level of difficulty

18 Kalman modeling Physiological response model Measurement model Kalman filter

19 Possible improvement of Kalman modeling Physiological response model Measurement model Kalman filter MACD filter for onset prediction