PLS’09 Beijing, China, September 7, 2009 Sponsored by US Army Research Office STIR: Advanced Estimation of Cognitive Status Contract No. W911NF-08-C SEP-2008 TO14-MAR-2009 PLS Tools in Electroencephalography Leonard J. Trejo PDT Institute Palo Alto, CA 94303, USA The 6 th International Conference on Partial Least Squares and Related Methods Sept. 4 th – 7 th, 2009 Beijing, China

PLS’09 Beijing, China, September 7, 2009 PDT and PDT Institute PDT Neuroergonomics Models and Applications Human-System Integration Human Performance Optimization Robust Biomedical Signal Processing Embedded and Real-time Systems for Bio-Sensing PDT Institute PhD/Masters/Undergraduate Training University Partners (UC Santa Cruz, Tsinghua University, UC San Diego, Univ. of West Florida)

PLS’09 Beijing, China, September 7, 2009 When I am not working…

PLS’09 Beijing, China, September 7, 2009 Outline Problem: stress, workload, fatigue and performance Response: Neuroergonomic models and control systems –Create useful definitions of cognitive states –Model, estimate and control cognitive states Background –Multimodal sensor-state models using PLS and KPLS Algorithms –Successes and failures: fatigue / BCI / engagement / workload New directions –Truly multidimensional sensor-process models –PARAFAC, N-PLS Summary

PLS’09 Beijing, China, September 7, 2009 Estimation of Cognitive States Other States Behavior and Performance Other Processes Internal Processes Biosignals

PLS’09 Beijing, China, September 7, 2009 Useful Definitions Engagement: selection of a task as the focus of attention and effort Workload: significant commit- ment of attention and effort to task Overload: task demands outstrip performance capacity Mental Fatigue: desire to with- draw attention and effort from a task General Cognitive Status Work- load Mental Fatigue Non- specific Factors Engage- ment

PLS’09 Beijing, China, September 7, 2009 Electroencaphalogram Cerebral Cortex the outermost layers of brain 2-4 mm thick (human)

PLS’09 Beijing, China, September 7, 2009 EEG Sources

PLS’09 Beijing, China, September 7, 2009 EEG Sources

PLS’09 Beijing, China, September 7, 2009 Other Elements of Sensor-State Models ModalityEffect of Workload Heart rateIncrease Heart rate variability (and HFQRS)Decrease Vertical and horizontal EOG (eye movements) Increase BlinksMay decrease for intake Pupil diameterIncrease Skin conductance, SCR, GSRIncrease EMG (frontalis, temporalis, trapezius)Increase

PLS’09 Beijing, China, September 7, 2009 Successful Application 1: Mental Fatigue Black = Alert Red = Mentally Fatigued Fz Pz Frontal Theta Parietal Alpha

PLS’09 Beijing, China, September 7, 2009

Successful Application 2: BCI

PLS’09 Beijing, China, September 7, 2009 Successful Application 2: BCI Central sulcusPrimary motor area Secondary motor area Lateral sulcus Primary motor area Resting StateReal or imaginary motion (Adapted from Beatty, 1995) EEG from Motor Areas

PLS’09 Beijing, China, September 7, 2009 Successful Application 2: BCI Control System for Target Practice Trial-by-trial classification (left, right) –250 ms display update Dual adaptive controller design –Adaptive PLS pattern recognition –Adaptive gain control for motion

PLS’09 Beijing, China, September 7, 2009 Successful Application 2: BCI

PLS’09 Beijing, China, September 7, 2009 Partly Successful Application: Mental Workload Estimation Trejo, et al. ACI 2007

PLS’09 Beijing, China, September 7, 2009 Stress, Workload, Fatigue and Performance Trejo, et al. ACI 2007

PLS’09 Beijing, China, September 7, 2009 Stabilizing Classifiers PLS Algorithm

PLS’09 Beijing, China, September 7, 2009 Multimodal Overload Patterns Time (s) Value

PLS’09 Beijing, China, September 7, 2009 Workload-related EEG Sources Passive viewing: theta alpha Engaged 5: theta alpha

PLS’09 Beijing, China, September 7, 2009 Application Summary Models of engagement, fatigue & BCI: % accurate Stable within a day Stable from day to day Models of mental workload: 60-90% accurate Moderately stable within a day Unstable from day to day

PLS’09 Beijing, China, September 7, 2009 Recommended Directions 1.Deployable multimodal sensors ( EEG, fNIR, EOG, gaze, HRV, EMG, SCR, SpO 2, BP, core body temperature, gesture, posture facial expression,...) 2.Multimodal experimental designs and operational tests 3.Advanced neurocognitive process models 4.Multimodal sensor-process mapping algorithms

PLS’09 Beijing, China, September 7, 2009 “Atomic” EEG Elements Atoms Molecule Basic Sources “atoms” Coherent Systems “molecules” Coherence Bonds Covalent Bonds

PLS’09 Beijing, China, September 7, 2009 “Molecular” EEG Processes Coherence Bonds Atoms

PLS’09 Beijing, China, September 7, 2009 Familiar (bilinear) Mapping Algorithms Factor Analysis Principal Component Analysis (PCA) afaf bfbf

PLS’09 Beijing, China, September 7, 2009 Multimodal Mapping How to generalize bilinear models to systems with more dimensions? 1.Unfolding a bilinear model a.Represent all experimental factors in one dimension b.Observations (trials) is second dimension c.Contrast each dimension vs. pairs of the other two 2.Multidimensional model a.Assume orthogonal factors: PARAFAC b.Allow interacting factors: Tucker 3 3.Modeling approach a.Unsupervised extraction: PARAFAC, CANDECOMP, Tucker 3 b.Supervised extraction: N-PLS

PLS’09 Beijing, China, September 7, 2009 Unfolding a Bilinear Model Unfolding Dim 1 Dim 2Dim 3 Dim 2 & 3 X X X1X1 X2X2 X3X3 Dim 1 & 3 Dim 1 & 2

PLS’09 Beijing, China, September 7, 2009 Multidimensional Modeling (Tucker 3 Model, unsupervised) x ijk is an element of (l x m x n) multidimensional array F1, F2, F3 are the number of components extracted on the 1st, 2nd and 3rd mode a, b, c are elements of the A, B, C loadings matrices for the 1st, 2nd and 3rd mode g are the elements of the core matrix G which defines how individual loading vectors in different modes interact e ijk is an error element (unexplained variance)

PLS’09 Beijing, China, September 7, 2009 PARAFAC ( Parallel Factor Analysis, unsupervised ) afaf bfbf cfcf PARAFAC is a special case of the Tucker 3 model where F1= F2 = F3=F and G = I For a 3-way array:

PLS’09 Beijing, China, September 7, 2009 N-way PLS (supervised) X frequency workload condition X afaf cfcf bfbf vfvf ufuf time max. covariance electrodes EEG Labels a f – spectral atom b f – spatial atom c f – temporal atom v f – workload atom u f – temporal atom

PLS’09 Beijing, China, September 7, 2009 Demo: Workload / PARAFAC EEG Workload condtions (e.g., trials, time) Electrodes EEG Frequency

PLS’09 Beijing, China, September 7, 2009 Summary Successes and Failures Fatigue, BCI, engagement: accurate, stable Workload: variably accurate, unstable Useful models of state-related EEG sources “Atomic” EEG sources “Molecular” EEG systems Approaches to multidimensional models and algorithms Tradtional bilinear methods (PCA, factor analysis, ICA) Truly multidimensional methods Correlated factors (Tucker 3) Uncorrleated factors (PARAFAC, CANDECOMP, N-PLS) Supervised algorithms (N-PLS)