Facial Affect Displays During Tutoring Sessions Mattijs Ghijsen – Dirk Heylen* Rieks op den Akker Anton Nijholt

2 INES

3 Facial Expressions What do they tell about how the student is experiencing the situation? How might we be able to use this to improve the tutoring system?

4 Previous Work Based on –Student Activity –Difficulty –Errors Adjust –Adjust teaching strategy –Adjust dialogue move

5 Pilot Collect data See what happens

6 Interpreting If some expression occurred What triggered it? What does it mean? = How can a system make sense of what happened and react appropriately?

7 Elements to consider Events: –system –student Appraisals Facial Expressions

8 Facial Expression Dictionary? Emotions Mental state Conversational function Adaptor

9 More complex Using Stimulus Evaluation Checks: –Situation: how could it be evaluated –Face: what could have triggered it Why do it? Some way of flexibility: –if match between f.e.-sec and sit.-sec: ok –if no match than: reason

10 Component Process Model Scherer Eclectic Appraisal process –Stimulus Evaluation Checks –5 criteria –Performed sequentially

11 Stimulus Evaluation Checks Noveltychange in the pattern of external or internal stimulation Intrinsic pleasantness whether a stimulus event is pleasant, inducing approach tendencies, or unpleasant, inducing avoidance tendencies Goal/Need significance whether a stimulus event is relevant to important goals or needs of the organism; outcome consistent with expectations; conducive or obstructive to reaching goals Coping potential evaluating the causation of a stimulus and the coping potential available, particularly the degree of control over the event and relative power Norm/Self compatibility evaluating whether the event, particularly an action conforms to social norms, cultural conventions…

12 Component Process Model FunctionSubsystemComponent Evaluation of stimulation Information processingCognitive System regulationSupportNeurophysiological Preparation and direction of action ExecutiveMotivational Communication and direction of action ActionExpressive Monitoring, attention, focus, reflection MonitorSubjective feeling

13 SEC  AU NoveltyHighLow Sudden /27 (1/2=brow raiser; 5= eyelid; 26/27=mouth) Familiarity4b + 7 Predictability4b + 7

14 INES

15 The Exercise Ask patient to put arm on table Disinfect part of the arm Insert the needle Inject medicine

16 Movie

17 Movie

18 Movie

19 Movie

20 Movie

21 Movie

22 Movie

23 Situation – Expression Patient asks for clarificationraised brows (2), frown (1), head pulled back (1) Patient repeatedly does not understand what student says smile (2) Patient pulls up her sleevesmile (4), raised brows (2), nod (1), pull head back (1) Student is disinfecting the armsmile (7), raised brow (1)

24 Facial Expression  Appraisal ExpressionNovelPleasantGoal significant Coping Potential Smile (22) Raised eyebrows (11) 7420 Mouth corners pulled down (2) 2111

25 Discussion The appraisal process is subjective and depends exclusively on the appraising individual’s perception of and inference about the characteristics of an event.

26 For our application Facial expressions tell us something that might be of use. –Students are finding it pleasant –Students are concentrating –Students are not frustrated –Students were not expecting something

27 Discussion Not only emotion Pilot: not real data Complicated procedure Many things indeterminate

28 Questions: How to classify situations in terms of SECs? –Situation: what to take into account? –Appraisal not situation –Coder consistency Facial expression: –how to analyse in terms of SECs –ambiguity – incompleteness How to make a system work with such information structures?

29 Problem Matching 1) potential appraisals of situation types with 2) potential significance of facial expression 1) = hard to do for types 2) = lots of ambiguity

30 So Why do it? Some way of flexibility: –if match between f.e.-sec and sit.-sec: ok –if no match than: reason