Computational Physiology Lab Department of Computer Science University of Houston Houston, TX Eustressed or Distressed? Combining Physiology with Observation in User Studies Avinash Wesley Dr. Peggy Lindner (Co-Advisor) Dr. Ioannis Pavlidis (Advisor)

2/13 Stress Signs Peripheral Physiological Measurement of Stress –Adrenergic response Elevates heart rate, respiration rate, and blood pressure –Cholinergic response Activates sweat glands on fingers and the perinasal area Introduction Methods Results and Discussion Acknowledgements  Stress Mechanism  Motivation  Background

3/13 Physiology and Observation Perspiratory response are –sympathetic in nature –Non-specific to positive or negative arousal Introduction Methods Results and Discussion Acknowledgements DistressEustress  Stress Mechanism  Motivation  Background

4/13 Emotions vs. Performance Introduction Methods Results and Discussion Acknowledgements  Stress Mechanism  Motivation  Background Arousal LOWMEDIUMHIGH Sleep Disorganization AnxietyAlertness Optimal An important goal in user studies: Study the role of emotions on human performance Emotions can be quantified via physiological response Physiological responses can be disambiguated via observation

5/13 Perspiration Signal and Observation Physiology –Perspiration extraction method in the Thermal Imagery [1] Observational Annotation –Traditional done in the Visual Imagery Manual Introduction Methods Results and Discussion Acknowledgements [1] D. Shastri, A. Merla, P. Tsiamyrtzis, and I. Pavlidis. Imaging facial signs of neurophysiological responses. IEEE Transactions on Biomedical Engineering, 56(2):477–484, Courtesy of Science channel  Stress Mechanism  Motivation  Background

6/13 Region Tracking Seven anatomical regions tracked over time by a dynamic template update tracker [2] Introduction Methods Results and Discussion Acknowledgements [2] Y. Zhou, P. Tsiamyrtzis, and I. Pavlidis. Tissue tracking in thermo-physiological imagery through spatio-temporal smoothing. Proc. of the 12th Int. Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2009),5762:1092–1099,  Facial Expression Recognition  Field Study

7/13 Pattern Classification Feature Vector Classifier – Classify five action units (AU1+2, 4, 9, 10, and 12) – Multilayer Perceptron – 10-fold Cross Validation Introduction Methods Results and Discussion Acknowledgements AU 1+2 Inner + Outer Eyebrow Raise d(x,5): Euclidean distance between ROI-x and 5, (x 5)  Facial Expression Recognition  Field Study

8/13 Surgical Training Surgeon Pool (n=17) –Novices –Experienced Tasks –Running string (Task-1) –Pattern cut (Task-2) –Intracorporeal suture (Task-3) Dataset: 977 Thermal Clips Introduction Methods Results and Discussion Acknowledgements  Facial Expression Recognition  Field Study

9/13 Validation Results Using Thermal Imagery –244 Facial Expressions –Ground Truth via Visual annotation –Method Accuracy 81.55% Introduction Methods Results and Discussion Acknowledgements * Use of visual images instead of thermal images for display purpose only * Confusion matrix  Quantitative Analysis  Qualitative Analysis  Conclusions

10/13 Results From The Field Study Distress is inversely related to experience –E N (Perinasal perspiratory signal on portions of negative emotions) Introduction Methods Results and Discussion Acknowledgements  Quantitative Analysis  Qualitative Analysis  Conclusions Novice Experienced

11/13 Example Visualizations Eustress Distress Introduction Methods Results and Discussion Acknowledgements  Quantitative Analysis  Qualitative Analysis  Conclusions

12/13 Conclusions The proposed method is –Comprehensive (quantitative and qualitative) –Economical (single imaging modality with no labor) Conducted a study design that is applicable to a broad class of Human Machine Interaction Future Work – Expand the facial expression set – Apply the method to more field studies Detection of pain onset-offset Introduction Methods Results and Discussion Acknowledgements  Quantitative Analysis  Qualitative Analysis  Conclusions

13/13 Support provided by NSF award # IIS Introduction Methods Results and Discussion Acknowledgements