1. Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho

2. Outline I.Experimental Setup II.Valence A.Problems Addressed B.Experimental Procedure C.Results D.Discussion III.Confusion A.Problems Addressed B.Experimental Procedure C.Results D.Discussion IV.Conclusions

3. Outline I.Experimental Setup II.Valence A.Problems Addressed B.Experimental Procedure C.Results D.Discussion III.Confusion A.Problems Addressed B.Experimental Procedure C.Results D.Discussion IV.Conclusions

4. Experimental Setup  Electrodes attached to: Corrugator supercilii (“brow”) Zygomaticus major (“cheek”)  Grounded on arm

5. Outline I.Experimental Setup II.Valence A.Problems Addressed B.Experimental Procedure C.Results D.Discussion III.Confusion A.Problems Addressed B.Experimental Procedure C.Results D.Discussion IV.Conclusions

6. Outline I.Experimental Setup II.Valence A.Problems Addressed B.Experimental Procedure C.Results D.Discussion III.Confusion A.Problems Addressed B.Experimental Procedure C.Results D.Discussion IV.Conclusions

7. Problems Addressed  Can we sense valence from facial electromyographic (EMG) activity? Intensity?  Can we sense valence from mouse behavior (pressure in particular)?  In general, what types of activity can we sense with facial EMG?

8. Relevant Prior Work  Recent study by Larsen, Norris, Cacioppo  Change in mean “cheek” EMG activity correlated positively with valence  Change in mean “brow” EMG activity negatively correlated with valence (CS = “brow” muscle, ZM = “cheek” muscle)

9. Outline I.Experimental Setup II.Valence A.Problems Addressed B.Experimental Procedure C.Results D.Discussion III.Confusion A.Problems Addressed B.Experimental Procedure C.Results D.Discussion IV.Conclusions

10. Experimental Procedure  6 subjects  General procedure: 1.Subject watches a short film clip 2.Subject answers questions about film content and experienced affective state  Subject uses pressure sensitive mouse  EMG output is recorded during all stages

11. Film Clip Sequence  Each film clip was roughly 3 minutes 1.Golf Instruction (Neutral) 2.Clip from “The Champ” (Negative)  Sadness in 94.2% (Gross & Levenson) 3.Golf Instruction (Neutral) 4.Clip from “Robin Williams Live” (Pos)  Amusement in 84.1% (Gross & Levenson)  Did not vary order

12. Outline I.Experimental Setup II.Valence A.Problems Addressed B.Experimental Procedure C.Results D.Discussion III.Confusion A.Problems Addressed B.Experimental Procedure C.Results D.Discussion IV.Conclusions

13. Self-reported Valence Consistent with Expectations  Neutral Valence: 4 subjects indicated “neutral” affect, 2 subjects indicated low intensity positive affect  Negative Valence: All 6 reported sadness  Positive Valence: 5 of 6 reported amusement The sixth subject did not like Robin Williams

14. Example of Output  Output normalized due to large scale differences

15. Mean Output Consistent with Prior Work

16. Individual Results Varied

17. Other Observations  For some subjects, brow activity was a good indicator of web form activity

18. Outline I.Experimental Setup II.Valence A.Problems Addressed B.Experimental Procedure C.Results D.Discussion III.Confusion A.Problems Addressed B.Experimental Procedure C.Results D.Discussion IV.Conclusions

19. Discussion  Highly sensitive to individual subject differences and electrode positioning  Mean output consistent with Larsen, Norris, Cacioppo  Do results extend to other positive/negative states?  Unable to correlate EMG output with self-reported mood intensity  EMG output may be useful as an activity recognition sensor  Unable to correlate mouse pressure or velocity with valence  Dynamic model of mouse behavior may produce better results?

20. Outline I.Experimental Setup II.Valence A.Problems Addressed B.Experimental Procedure C.Results D.Discussion III.Confusion A.Problems Addressed B.Experimental Procedure C.Results D.Discussion IV.Conclusions

21. Outline I.Introduction and Experimental Setup II.Valence A.Problems Addressed B.Experimental Procedure C.Results D.Discussion III.Confusion A.Problems Addressed B.Experimental Procedure C.Results D.Discussion IV.Conclusions

22. Problem Addressed  Does the facial expression ‘frowning’ represent the feeling of confusion?  Can we recognize the feeling of confusion with the EMG outputs?  Does the importance of understanding influence the feeling of confusion?

23. Outline I.Introduction and Experimental Setup II.Valence A.Problems Addressed B.Experimental Procedure C.Results D.Discussion III.Confusion A.Problems Addressed B.Experimental Procedure C.Results D.Discussion IV.Conclusions

24. Experimental Procedure  6 subjects  General procedure: 1.Subject listens to an audio clip 2.Subject answers questions about audio content and experienced affective state  EMG output is recorded during all stages

25. Eliciting Confusion with Audio Recordings AB Mood control40 s classic music Controlspoken with standard American English Confused by accentspoken with accent Confused by meaning A brief paragraph with confusing content spoken with American English Two levels of the importance of understanding:  Low level importance of understanding.  High level importance of understanding. Audio clips to induce different levels of confusion:

26. Ratings and Measurements for feeling of Confusion  Subjective rating 5 scale Self-report confusion rating  Objective rating Test of understanding  Measurement EMG responses

27. Outline I.Introduction and Experimental Setup II.Valence A.Problems Addressed B.Experimental Procedure C.Results D.Discussion III.Confusion A.Problems Addressed B.Experimental Procedure C.Results D.Discussion IV.Conclusions

28. Result: EMG responses Brow Cheek Mean210.142654 Std214.445876 Difference in importance of understanding NO DF=1, P=0.28 NO DF=1, P=0.24

29. Result: Average EMG responses For brow  More EMG activity when  filling the web form  listening to clips with accent  EMG in ‘web-filling’ parts increase with level of confusion in high importance of understanding For cheek  EMG activity increase with degree of confusion ->Subject started to laugh

30. Result: Self confusion rating and test score create the feeling of confusion consistently Self rating confusion Test score Mean3.812.38 Std1.511.87 Difference for 6 clips Yes DF=5, P < 0.0001 Yes DF=5, P < 0.0001 difference among subjects NO DF=5, P=0.42 NO DF=5, P=0.33 Average plot

31. Outline I.Introduction and Experimental Setup II.Valence A.Problems Addressed B.Experimental Procedure C.Results D.Discussion III.Confusion A.Problems Addressed B.Experimental Procedure C.Results D.Discussion IV.Conclusions

32. Discussion  Huge individual difference in EMG responses -> normalize the data  More Brow activity when filling web forms -> It is more correlated to ‘the process of thinking about confusion thing’  More Brow activity when listening to clip with accent  Cheek activity increase with level of confusion -> So confused that subjects were giving up  Difference in importance of understanding

33. Outline I.Introduction and Experimental Setup II.Valence A.Problems Addressed B.Experimental Procedure C.Results D.Discussion III.Confusion A.Problems Addressed B.Experimental Procedure C.Results D.Discussion IV.Conclusions

34. Conclusions  Baseline difference in EMG for different muscles  Individual difference in EMG response  EMG response sensitive to electrode positioning  Brow EMG is negatively correlated with valence and positively correlated with feeling of confusion  Cheek EMG is positively correlated with valence  Facial EMG may be useful for activity recognition