Titel van de presentatie 16-9-2019 16:44 A tool to estimate STRESS (MENTAl state) in real time Anne-Marie Brouwer Maarten Hogervorst Rob van de Pijpekamp Jan van Erp

BACKgrond: estimating mental state using neurophysiology Titel van de presentatie 16-9-2019 16:44 BACKgrond: estimating mental state using neurophysiology Estimating mental (affective or cognitive) state of one individual at one moment in time (Neuro)physiological effects of mental processes and not caused by confounds EEG patterns looks different for difficult than easy scenarios, because: mental stress differs or: .. more movements, causing motion artefacts .. more visual information processing, leading to more active visual cortex .. Almost 10 years ago, we started working on bci: controlling computers.. From there we moved to monitoring mental state from bio signals. I will shortly describe a study that shows how we investigate the ‘real’ relation between neurophysio and mental processes, so that we know which variables actually carry information about the mental state under study, and then I describe an approach, or tool, that may be useful for out of the lab situations and is built on the methods used in the lab study.

LAB APPROACH: N-Back taSk Titel van de presentatie 16-9-2019 16:44 LAB APPROACH: N-Back taSk Difficulty varies across conditions while visual input and movement remain the same ‘Is the presented letter a target or not?’ 0-1-2 back in 2- minute blocks 500 ms 2000 ms 0-back-target 1-back-target 2-back-target

Titel van de presentatie 16-9-2019 16:44 classification Individually trained model (support vector machine) Training on first part of data, testing on last (unseen) part of data

Relative sensitivity of different measures Titel van de presentatie 16-9-2019 16:44 Relative sensitivity of different measures 2-vs-0-back, 2 minutes of data EEG: Most measures around 85% correct One electrode suffices Physiology: Skin conductance, ECG around significance level (60%) Breathing frequency: 68% Eye: Blinkduration: 50%, blink rate: 63%, Pupilsize: 75%

Titel van de presentatie 16-9-2019 16:44 CombinatiON EEG: 85% Physiology: 75% Eye: 75% All sensors: 91% Hogervorst, Brouwer and van Erp JBF (2014) Combining and comparing EEG, peripheral physiology and eye-related measures for the assessment of mental workload. Frontiers in Neuroscience 8:322

Titel van de presentatie 16-9-2019 16:44 IF we want to apply this.. Generalization across situations Generalization across subjects (training data) Deal with movements and confounds Idea of potential variables of interest

Titel van de presentatie 16-9-2019 16:44 ESTIMERE Tool to estimate mental state in real time, out of the lab No worries about confounds: ‘get all information that is possibly out there’ Automatically adapts to changing situation Met Estimere willen we nu real time en in praktijksituatie mental state inschatten. Praktijksituaties: altijd sprake van confounds. Die kun je echter ook als informatiebron behandelen. Als beweging en werklast voor een bepaald scenario altijd gecorreleerd zijn kun je daar gebruik van maken. Waarbij dan de vraag is of je die beweging het meest handig via ruis in EEGkanalen kunt meten of op een andere manier.

Titel van de presentatie 16-9-2019 16:44 CURRENT setup Pupilsize Blinkfrequency Eyelid opening Heart rate Estimated stress Classifier mio Keyboard hits mouse movements Match? Ground truth: Subjectively experienced stress level

Titel van de presentatie 16-9-2019 16:44 CURRENT setup

Titel van de presentatie 16-9-2019 16:44 Flexible system Easy to adapt: Type of mental state Sensors Variables Type of classification model Selection of retraining data Frequency of the probe question Another type of ground truth …

Titel van de presentatie 16-9-2019 16:44 tool To monitor users for adaptive automation or evaluation For research: How well can we estimate certain states In certain situations? Which sensors and variables are most valuable? How well does it generalize over persons and situations? Tool to monitor users for adaptive automation or evaluation

Titel van de presentatie 16-9-2019 16:44 Lab -> real life Pick an ideal mental state monitoring case: Few body movements Added value: patients that do not/cannot convey information well in other ways Research in the same context as the application 1.12 Non-Emotional Emotional Few body movements, at least at time of measurement/interpretation Omzeilen van generalisatie issue: je traint het model met data zo dicht mogelijk bij de situatie en persoon ‘of interest’ 1.08 R to R interval 1.04 1.00 Brouwer, Hogervorst, Reuderink, van der Werf, van Erp (2015) Physiological signals distinguish between reading emotional and non-emotional sections in a novel. Brain-Computer Interfaces, 1-14. 20 40 60 80 100 120 Page number