1. AAMAS 2003 CARTE @ USC / Information Sciences Institute Social Interaction with Agents Lewis Johnson Director, CARTE USC / Information Sciences Institute

2. AAMAS 2003 CARTE @ USC / Information Sciences Institute Jeff Rickel 1963 - 2003

3. AAMAS 2003 CARTE @ USC / Information Sciences Institute Rationale: Reeves & Nass’s Media Equation n People tend to relate to computers and other media as they do to people n Confirmed by multiple experimental studies n Synthetic agents exploit this tendency

4. AAMAS 2003 CARTE @ USC / Information Sciences Institute Claims n Synthetic agents raise expectations of ä Ability to understand user’s activities ä Social interaction skills, i.e., social intelligence n The challenge: to meet these expectations ä Primary focus: pedagogical agents ä Educational context helps constrain the problems ä Implications for other types of human-agent interaction

5. AAMAS 2003 CARTE @ USC / Information Sciences Institute Social Intelligence Implies: n Ability to model people, other agents ä May include goals, plans, emotions, motivations, personality, etc. n Ability to engage in social interaction ä Sensitive to the model of the person or agent ä Sensitive to the social context ä In coordination with task activities ä To motivate, influence, develop rapport

6. AAMAS 2003 CARTE @ USC / Information Sciences Institute Adele Pedagogical Agent

7. AAMAS 2003 CARTE @ USC / Information Sciences Institute Some Failures in Social Intelligence n Criticizing the same mistake over and over n Interrupting the learner after minor mistakes n Giving impression of negative emotional reactions to learner’s actions n Failing to show respect for the learner’s work n Failing to offer encouragement when the learner needs it n Failing to offer help when the learner is confused and frustrated

8. AAMAS 2003 CARTE @ USC / Information Sciences Institute Teasing Apart Social Intelligence

9. AAMAS 2003 CARTE @ USC / Information Sciences Institute One Aspect of SI: Team Coordination n Stems from an agent-oriented view of human-computer interaction n Learner, virtual tutor, other agents act together as a team n Roles and responsibilities dynamically allocated among team members n Learning activities, teamwork, and task work are flexibly integrated n Examples: Steve, MRE

10. AAMAS 2003 CARTE @ USC / Information Sciences Institute Steve Team demo

11. AAMAS 2003 CARTE @ USC / Information Sciences Institute Mission Rehearsal Exercise

12. AAMAS 2003 CARTE @ USC / Information Sciences Institute Steve/MRE Team/Task Model n Team task represented as a hierarchical, nonlinear plan ä Including possible alternative courses of action ä Courses of action evaluated for expected utility n Step responsibilities (and authorities) assigned to team members ä Possibility of dynamic sharing of responsibilities n Each team member has own (possibly partial) model of team plan & status Rickel & Johnson, IJCAI ‘01; Traum, Rickel, Gratch, & Marsella, AAMAS ‘03 See also: Scerri et al., AAMAS ’03; Davies et al., IUI ‘01

13. AAMAS 2003 CARTE @ USC / Information Sciences Institute Robot-Agent-Person Teams n Each team member supported by proxy ä Aids in team coordination, communication, role (re)allocation ä Supports adjustable autonomy ä Enables training & performance support n Test bed: disaster rescue simulation ä Human fire chief collaborates with simulated fire brigades to fight fires Information agents Team proxy Team Oriented Program Scerri, Pynadath, Johnson, Rosenbloom, Si, Schurr, Tambe: AAMAS ‘03

14. AAMAS 2003 CARTE @ USC / Information Sciences Institute Another Aspect of SI: Face-to- Face Communication n Example: MRE n Dialog model tracks state of communication among team members n Components: ä Which team members are in contact ä Who and what is being attended to ä Who are participants (or overhearers) in the conversation ä Who has the conversational initiative ä Common knowledge  Including social commitments to actions and to facts ä Communicative acts and grounding acts ä Negotiation acts ä Eye gaze, nonverbal gestures signal attention, grounding Marsella, et al., 2003; Traum & Rickel, AAMAS ’02; Traum et al., AAMAS ‘03 Dialog demo

15. AAMAS 2003 CARTE @ USC / Information Sciences Institute Another Aspect of SI: Interaction Tactics n How to help the learner ä Respecting learner’s autonomy & sense of control n How to influence the learner ä Motivating the learner as needed n When not to help the learner ä Reinforcing autonomy, engagement n Assume appropriate social stance toward the learner ä Interaction in the context of a social relationship n Bottom line: ä Influence of social relationships on human-agent interaction ä Rhetoric for human-agent interaction

16. AAMAS 2003 CARTE @ USC / Information Sciences Institute Example: Carmen’s Bright IDEAS Marsella, Johnson, & LaBore: Agents 2000, AI-Ed ‘03

17. AAMAS 2003 CARTE @ USC / Information Sciences Institute Gina’s Dialog Model Suggest Strategy Prompt Next Step Summarize Answer Question Reassure Praise n Gina’s main struggle: Get Carmen thru the I-D-E-A-S steps ä At each step, suggest a joint strategy (e.g. “old 5Ws”) ä Prompt/motivate Carmen thru that strategy ä React to Carmen’s emotional and cognitive state ä Employ interaction tactics based upon Carmen’s state; some focus on cognitive state, others on emotional state

18. AAMAS 2003 CARTE @ USC / Information Sciences Institute Questions About the “Gina Model” n How appropriate is it for educational applications? ä It is based on clinicians’ counseling-oriented view of training ä It is a dramatization of instructional interaction  Built from a deconstructed script n Empirical studies of tutorial interaction were needed ä To see how this model applies to other educational settings ä To determine which learner characteristics are most relevant ä To study social interaction processes in such settings

19. AAMAS 2003 CARTE @ USC / Information Sciences Institute Social Intelligence Project n Develop model of socially intelligent interaction for educational software ä Sensitive to important learner conative states, characteristics ä Sensitive to the social relationship with the learner ä To determine when to interact with the learner ä To determine how to interact with the learner n Application: Virtual Factory Teaching System ä Builds on existing Automated Laboratory Instructor (ALI) ä Originally designed for advanced industrial engineering students

20. AAMAS 2003 CARTE @ USC / Information Sciences Institute Experimental Study n Videotaped sessions of computer- based learning with human tutors, over multiple sessions n Students read tutorial on line and perform series of exercises with Virtual Factory Teaching System Johnson, Pain, Shaw, et al: IUI ’03, AIEd ‘03

21. AAMAS 2003 CARTE @ USC / Information Sciences Institute Conclusions from Study n Wide variation in learners’ preferred interaction styles ä Some prefer collaboration, some prefer working alone n Wide variation in confidence ä Between subjects ä Over time n Tutor generally able to assess learner confidence, ability, preferred interaction style

22. AAMAS 2003 CARTE @ USC / Information Sciences Institute Conclusions from Study (Cntd.) n Information used by tutor: ä Expectations from knowledge of task ä Eye gaze, mouse location ä Verbal feedback from student

23. AAMAS 2003 CARTE @ USC / Information Sciences Institute Techniques for Promoting Learner Engagement n Tutor phrased comments in order to reinforce learner control and joint activity. E.g.: ä “Why don’t you go ahead and read your tutorial factory” ä “You want to save the factory” ä “I’d skip this paragraph” ä “So why don’t we do that?” n Tutor avoided giving direct instructions ä Except for operating the interface

24. AAMAS 2003 CARTE @ USC / Information Sciences Institute Interaction Tactics n Tutor and student communicate via interaction tactics ä Tactics frequently fail first time, requiring rephrasing ä Tactic not considered complete unless desired effect is achieved. E.g.: Tutor: So it’s asking for regression Student: Right, that wasn’t an option… there’s no place… Tutor: You want to click on regression here…

25. AAMAS 2003 CARTE @ USC / Information Sciences Institute Theoretical Framework: Learner Motivation n Motivational factors ä “Four Cs”:  Curiosity  Challenge  Confidence  Control ä Learner goals and meta-goals:  Persistent goals  Attitudes toward goal achievement

26. AAMAS 2003 CARTE @ USC / Information Sciences Institute Role of Motivational Factors n Curiosity ä Employ tactics that promote inquiry n Challenge ä Select tasks according to difficulty ä Intervene in response to learner confusion, hard impasses n Confidence ä Regulate amount of feedback n Control ä Employ tactics that promote learner goal-setting n Learner goals ä Employ tactics that promote learner goal identification

27. AAMAS 2003 CARTE @ USC / Information Sciences Institute Theoretical Framework: Politeness (Brown & Levinson) n Social actors motivated by face wants ä Negative face: freedom of action and freedom from imposition; autonomy ä Positive face: consistent self-image, and desire that self- image is appreciated and approved of by others n Face-threatening acts pervasive in interaction ä Warnings, offers, promises, challenges, emotional displays n Face threat depends upon power, distance, ranking of threats due to social context n Social actors employ politeness tactics to mitigate face threat

28. AAMAS 2003 CARTE @ USC / Information Sciences Institute Role of Politeness Factors in Tutorial Interaction n Common tutorial actions (advice, hints) are face-threatening acts n Tactic failures impact agent’s positive face n Face threat depends upon distance ä Distance depends on duration of interchange, established trust, learner’s negative face wants (preference for autonomy vs. collaboration) n Choose tactics to promote learner positive face, mitigate negative face threat ä By promoting shared goals ä By avoiding direct instructions ä By reinforcing positive (self-)assessment of goal achievement ä When dictated by social distance, learner motivational factors

29. AAMAS 2003 CARTE @ USC / Information Sciences Institute Example Interaction Tactics n Rhetorical requests to give hints ä “Can I give you a hint? Try this…” ä Question reinforces learner negative face; failure to wait for answer avoids positive face threat n Hints phrased as questions ä “Do you want to do x?” ä Reinforces learner control (positive face), can influence learner goals (positive face) n Hints as suggestions ä “You could do x.” ä Similar face effects as questions

30. AAMAS 2003 CARTE @ USC / Information Sciences Institute Interaction Tactics (Cntd.) n Hints as suggestions of joint goal ä “Let’s do x.” ä Suggestion mitigates negative face threat; reference to joint goal influences positive face wants; depends on learner autonomy preferences n Hints as references to tutorial authors ä Deflect blame for face threat to authors n Imperative hints ä Used only when blame is deflected (I.e., to interface), or possibly when distance is reduced

31. AAMAS 2003 CARTE @ USC / Information Sciences Institute SI Text Generator n Generates text for interaction tactics ä Input: type of intervention, object(s) of intervention, style of reference (e.g., as joint goal, user’s goal, etc.) ä Parameters: social distance: importance of motivational influence ä Common tutor wording styles captured and codified ä Wording style chosen randomly if not specified

32. AAMAS 2003 CARTE @ USC / Information Sciences Institute Architecture of SI System

33. AAMAS 2003 CARTE @ USC / Information Sciences Institute Tracking Learner Attention & Confusion n Useful: ä To detect proactive interaction opportunities  Avoids learner frustration ä To avoid inappropriate interruptions  Avoids negative effects on learner affect, trust ä To assess learner engagement  Helps determine objectives for interaction tactics

34. AAMAS 2003 CARTE @ USC / Information Sciences Institute Assessing Learner Attention: Methods n Track learner’s interactions with tutorial and simulation interface n Track learner gaze n Fuse using Bayesian techniques to determine focus n Instrument tutorial with expected learner goals, time demands n Infer overall learner activity (e.g., scan vs. problem-solve), overall engagement, specific impasses

35. AAMAS 2003 CARTE @ USC / Information Sciences Institute Student Interface

36. AAMAS 2003 CARTE @ USC / Information Sciences Institute Student Interface with Motivational Self-Assessment

37. AAMAS 2003 CARTE @ USC / Information Sciences Institute Data Collection Method n Client classes for reporting events and states ä C# implementation ä Java SOAP interface n Reporting classes send data to Web Service running on server n Web Service organizes and saves in database on server

38. AAMAS 2003 CARTE @ USC / Information Sciences Institute VFTS Plans

39. AAMAS 2003 CARTE @ USC / Information Sciences Institute Inferring Learner Focus

40. AAMAS 2003 CARTE @ USC / Information Sciences Institute Tracking Focus of Attention

41. AAMAS 2003 CARTE @ USC / Information Sciences Institute A Final Comment: Social Actors or Dramatic Actors? n Social actor view: ä Interaction with agent is a social interaction ä Agent should act in a manner consistent with human social interaction ä Inspiration: theories of social interaction n Dramatic actor view: ä Interaction is part of an unfolding story ä Agent should act so as to contribute to the story and its message  Make action clear, understandable, and engaging ä Inspiration: theories of drama and narrative n Intersection: ä Social theories of presentation of self – e.g. Goffman ä Relationship between rhetoric and drama in effective communication