PLS’09 Beijing, China, September 7, 2009 Sponsored by US Army Research Office STIR: Advanced Estimation of Cognitive Status Contract No. W911NF-08-C SEP-2008 TO14-MAR-2009 PLS Tools in Electroencephalography Leonard J. Trejo PDT Institute Palo Alto, CA 94303, USA The 6 th International Conference on Partial Least Squares and Related Methods Sept. 4 th – 7 th, 2009 Beijing, China

PLS’09 Beijing, China, September 7, 2009 PDT and PDT Institute PDT Neuroergonomics Models and Applications Human-System Integration Human Performance Optimization Robust Biomedical Signal Processing Embedded and Real-time Systems for Bio-Sensing PDT Institute PhD/Masters/Undergraduate Training University Partners (UC Santa Cruz, Tsinghua University, UC San Diego, Univ. of West Florida)

PLS’09 Beijing, China, September 7, 2009 When I am not working…

PLS’09 Beijing, China, September 7, 2009 Outline Problem: stress, workload, fatigue and performance Response: Neuroergonomic models and control systems –Create useful definitions of cognitive states –Model, estimate and control cognitive states Background –Multimodal sensor-state models using PLS and KPLS Algorithms –Successes and failures: fatigue / BCI / engagement / workload New directions –Truly multidimensional sensor-process models –PARAFAC, N-PLS Summary

PLS’09 Beijing, China, September 7, 2009 Estimation of Cognitive States Other States Behavior and Performance Other Processes Internal Processes Biosignals

PLS’09 Beijing, China, September 7, 2009 Useful Definitions Engagement: selection of a task as the focus of attention and effort Workload: significant commitment of attention and effort to an engaged task Visual, Auditory, Haptic Psychomotor Cognitive (memory, executive) Overload: task demands outstrip performance capacity Mental Fatigue: desire to withdraw attention and effort from an engaged task associated with extended performance (~45 min) General Cognitive Status Work- load Mental Fatigue Non- specific Factors Engage- ment

PLS’09 Beijing, China, September 7, 2009 Electroencaphalogram Cerebral Cortex the outermost layers of brain 2-4 mm thick (human)

PLS’09 Beijing, China, September 7, 2009 EEG Sources A pyramidal neuron with a soma, apical & basal dendrites and a single axon

PLS’09 Beijing, China, September 7, 2009 EEG Sources A pyramidal neuron with a soma, apical & basal dendrites and a single axon

PLS’09 Beijing, China, September 7, 2009 Other Elements of Sensor-State Models ModalityEffect of Workload Heart rateIncrease Heart rate variability (and HFQRS)Decrease Vertical and horizontal EOG (eye movements) Increase BlinksMay decrease for intake Pupil diameterIncrease Skin conductance, SCR, GSRIncrease EMG (frontalis, temporalis, trapezius)Increase

PLS’09 Beijing, China, September 7, 2009 Successful Application 1: Mental Fatigue Black = Alert Red = Mentally Fatigued Fz Pz Frontal Theta Parietal Alpha

PLS’09 Beijing, China, September 7, 2009

Successful Application 2: BCI

PLS’09 Beijing, China, September 7, 2009 Successful Application 2: BCI

PLS’09 Beijing, China, September 7, 2009 Stress, Workload, Fatigue and Performance Trejo, et al. ACI 2007

PLS’09 Beijing, China, September 7, 2009 Cognitive Overload (Trejo, et al. ACI 2007) Trejo, et al. ACI 2007

PLS’09 Beijing, China, September 7, 2009 Stabilizing Classifiers

PLS’09 Beijing, China, September 7, 2009 Multimodal Overload Patterns Time (s) Value

PLS’09 Beijing, China, September 7, 2009 Workload-related EEG Sources Passive viewing: theta alpha Engaged 5: theta alpha

PLS’09 Beijing, China, September 7, 2009 Application Summary Individual models of engagement and fatigue: % accurate Stable within a day Stable from day to day Individual models of workload or effort: 60-90% accurate Moderately stable within a day Unstable from day to day Normative models (limited data): 50-70% accurate Moderately stable

PLS’09 Beijing, China, September 7, 2009 Recommended Directions 1.Deployable multimodal sensors ( EEG, fNIR, EOG, gaze, HRV, EMG, SCR, SpO 2, BP, core body temperature, gesture, posture facial expression,...) 2.Multimodal experimental designs and operational tests 3.Advanced neurocognitive process models 4.Multimodal sensor-process mapping algorithms APECS

PLS’09 Beijing, China, September 7, 2009 “Atomic” Decomposition “In the parlance of modern harmonic analysis (Chen and Donoho, 2001), we performed a space/ frequency/time ‘‘atomic decomposition’’ of multidimensional data. In other words, we assume that each neural mass contributes a distinctive atom to the topographic frequency/time description of the EEG, so that the estimation of these atoms is possible by means of signal- processing techniques. Each atom will be defined by its topography, spectral content, and time profile; in other words, by its spatial, spectral, and temporal signatures.” Fumikazu Miwakeichi, et al, Decomposing EEG data into space–time–frequency components using Parallel Factor AnalysisNeuroImage 22 (2004) 1035–1045. Chen, S., Donoho, D., Atomic decomposition by basis pursuit. SIAM Rev. 43, 129– 159.

PLS’09 Beijing, China, September 7, 2009 “Atomic” EEG Elements Atoms Molecule Basic Sources “atoms” Coherent Systems “molecules” Coherence Bonds Covalent Bonds

PLS’09 Beijing, China, September 7, 2009 “Molecular” EEG Processes Coherence Bonds Atoms

PLS’09 Beijing, China, September 7, 2009 Familiar (bilinear) Mapping Algorithms Factor Analysis Principal Component Analysis (PCA) afaf bfbf

PLS’09 Beijing, China, September 7, 2009 Multimodal Mapping How to generalize bilinear models to systems with more dimensions? 1.Unfolding a bilinear model a.Represent all experimental factors in one dimension b.Observations (trials) is second dimension c.Contrast each dimension vs. pairs of the other two 2.Multidimensional model a.Assume orthogonal factors: PARAFAC b.Allow interacting factors: Tucker 3 3.Modeling approach a.Unsupervised extraction: PARAFAC, CANDECOMP, Tucker 3 b.Supervised extraction: N-PLS

PLS’09 Beijing, China, September 7, 2009 Unfolding a Bilinear Model Unfolding Dim 1 Dim 2Dim 3 Dim 2 & 3 X X X1X1 X2X2 X3X3 Dim 1 & 3 Dim 1 & 2

PLS’09 Beijing, China, September 7, 2009 Multidimensional Modeling (Tucker 3 Model, unsupervised) x ijk is an element of (l x m x n) multidimensional array F1, F2, F3 are the number of components extracted on the 1st, 2nd and 3rd mode a, b, c are elements of the A, B, C loadings matrices for the 1st, 2nd and 3rd mode g are the elements of the core matrix G which defines how individual loading vectors in different modes interact e ijk is an error element (unexplained variance)

PLS’09 Beijing, China, September 7, 2009 PARAFAC ( Parallel Factor Analysis, unsupervised ) afaf bfbf cfcf PARAFAC is a special case of the Tucker 3 model where F1= F2 = F3=F and G = I For a 3-way array:

PLS’09 Beijing, China, September 7, 2009 N-way PLS (supervised) X frequency workload condition X afaf cfcf bfbf vfvf ufuf time max. covariance electrodes EEG Labels a f – spectral atom b f – spatial atom c f – temporal atom v f – workload atom u f – temporal atom

PLS’09 Beijing, China, September 7, 2009 Demo: Workload / PARAFAC EEG Workload condtions (e.g., trials, time) Electrodes EEG Frequency

PLS’09 Beijing, China, September 7, 2009 Summary Successes and Failures Individual models of engagement and fatigue: accurate, stable Individual models of workload or effort: variably accurate, unstable Normative models (limited data): inaccurate, unstable Useful models of state-related EEG sources “Atomic” EEG sources “Molecular” EEG systems Approaches to multidimensional models and algorithms Tradtional bilinear methods (PCA, factor analysis, ICA) Truly multidimensional methods Correlated factors (Tucker 3) Uncorrleated factors (PARAFAC, CANDECOMP, N-PLS) Supervised algorithms (N-PLS)

PLS’09 Beijing, China, September 7, 2009 Within-day Results