1. Physiological sensors and EEG A short introduction to (neuro-)physiological measurements

2. Outline 1. Physiological sensors and EEG 2. Classification of affective states 2.1 Background & experiment design 2.2 Features, Classification and Resumee

3. The sensors Electroencephalogram (EEG) Electroocculogram (EOG) Electromyogram (EMG) Electrocardiogram (ECG) Galvanic skin response (GSR) Blood volume pulse (BVP) Respiration Temperature central nervous system peripheral nervous system

4. Electro-----gram..occulo....myo....cardio..

5. Blood volume pulse infrared photoelectric sensor detects changes in tissue blood volume 60 – 80 pulses a minute > heart rate and inter-beat-interval can be derived faster when exercised or aroused

6. Galvanic skin response impedance of the skin is measured eccrine glands at hands and food soles increases linear with arousal

7. Respiration circumference of chest is measured via strain gauge normal breath rate: 12 - 16 breath per minute (~500ml) faster and more shallow with higher arousal

8. Electroencephalogram normal signal theta band alpha band beta band gamma band 0 – 1000 Hz 4 – 8 Hz 13 – 30 Hz 8 – 12 Hz 30 – 100 Hz

9. The 10 – 20 system

10. The lobes of the brain

11. Time- vs frequency domain normal signal theta band alpha band beta band gamma band 0 – 1000 Hz 4 – 8 Hz 13 – 30 Hz 8 – 12 Hz 30 – 100 Hz

12. Event-related potentials (ERPs) The raw signal with stimuli presentations The averaging of the raw signal epochs leading to the ERP

13. ERPs from affective pictures

14. Time-frequency analysis evoked frequency analysis induced frequency analysis Tallon-Baudry & Bertrand, 1999

15. Recapitulation EEG measures the electrical potentials from synchroneously active pyramidal cells in the cortex high temporal resolution vs low spatial resolution only weak signals from deeper structures & closed fields “blindness” ERP analysis in time domain shows mainly low frequency components analysis in frequency domain (evoked vs induced) also pics up high frequency components

16. Classification of affective states Background & experiment design

17. Motivation Affective Computing aimes at the enrichment of human- computer interaction Physiological and neurophysiological signals give access to (non-observable) affective and cognitive states (Sander et al. 2005)

18. Emotion Models 2 dimensional model of emotion valence arousal posneg low high boredom joy fear relaxation An emotion, or an affective state, is a reaction to an internal (e.g. thought) or external event (e.g. visual or auditory stimuli), and a behavioural disposition. frustration

19. Regulation Affective State Appraisal Stimulus presentation +/- Hippocampus (HC) dorsal Anterior Cingulate Gyrus (dACG) dorsal prefrontal cortex (PFC) Amygdala Insula ventral Anterior Cingulate Gyrus (vACG) ventral prefrontal cortex (PFC) Inside Outside HC Insula Amygdala dACG vACG PFC Neurobiology of Emotional Perception Phillips et al. 2003

20. Overview Issues aBCI Elicitation of emotions Ground truth construction Sensor modalities Modality fusion Feature selection and reduction Classification

21. Example Study

22. Elicitation of emotions not easy to establish a natural context of emotion elicitation: subject- vs event-elicited lab setting vs. real world expression vs feeling open vs hidden recording emotion vs other purpose (for subject)

23. Affective Stimuli valence arousal posneg low high boredom joy fear relaxation frustration

24. Ground truth construction normed stimulus sets (e.g. IAPS, IADS) versus self-assessment after each trial

25. Sensor modalities physiological modalities: slow response to stimulus (seconds) long inter-trial intervals (seconds) few (but long presented) stimuli per subject neurophysiological modality: –fast response to stimulus (miliseconds) –short inter-trial intervals (miliseconds) –many (but short presented) stimuli per subject

26. Experiment: Design Resting Stimulus SAM rating 3 – 4 s 6 s not limited 1. emotion induction with visual affective stimuli 2. self-assessment of (induced) affective state after each stimulus with Self-Assessment Maneken (SAM)

27. Classification of affective states Features, Classification and Resumee

28. Modality fusion data-level fusion feature-level fusion decision-level fusion

29. Feature selection and reduction Selection: manual selection (e.g. literature research) automatical selection / “wrapper” method Reduction: mapping from high into low dimensional space (e.g. PCA,ICA) > “filter” method

30. Feature extraction EEG: 6 features (from 6 frequency bands over 6 brain regions) physiological Sensors: mean, variance, and (minimum & maximum) from GSR, blood pressure, heart rate, respiration, and temperature

31. Classification many possible classification approaches: decision trees, linear discriminant analysis, support vector machine, neural networks… optimal method depends on structure of data !keep training trials appart from test data, to avoid the contamination of the classifier.. !

32. Results 2 classes: high vs low arousal 3 classes: high, medium and low arousal

33. Classification physiological studies: 4 class: up to 95% 8 class: 81% (!) generalization of classifiers over subjects and time neurophysiological studies: –3 class: up to 67% –2 class: up to 79% –? generalization of classifiers over subjects and time ?

34. Recapitulation natural context of emotion elicitation characteristics of sensors are important many different approaches for data fusion, feature selection/reduction and classification – no optimal method per se still a long way toward an affect classification in an natural and multimodal real-world setting.. but we are getting there!

35. Thanks