1. Embodied Conversational Agents: A Case Study of Freudbot Bob Heller, PhD Athabasca University November 3, 2004

2. Mike Proctor – AIML programmer Dean Mah – Web implementation Billy Cheung – Graphics, test chatter Lisa Jewell – Chat log analysis, content developer, test chatter Julianna Charchun – Chat log analysis Jude Onuh – AIML programmer Acknowledgements

3. Embodied Conversational Agents Definitions Embodiment in Conversational Interfaces: REA (Cassel et al., 1999) Embodied Conversational Agents (Cassel, Sullivan, Prevost, & Churchill, 2000) –FMTB model Vos (2002) offers 5 features of ECA –Human like appearance –Body used for communication purposes –Natural communication protocols –Multimodality –Social role

4. Embodied Conversational Agents Anthropomorphic Agents Animated Interface Agents Animated Pedagogical agents Pedagogical Agent Persona Intelligent Tutoring Systems -AutoTutor (Graesser et al) http://www.autotutor.org/index.htm http://www.autotutor.org/index.htm Chatterbots or Chatbots - Weizenbaum’s (1966) Eliza

5. Why? -primacy of conversation -Constructivist theory -The Media Equation -Persona effect -cognitive load Embodied Conversational Agents

6. Richard Wallace and A.L.I.C.E. Artificial Linguistic Internet Computer Entity http://alicebot.org/ 3 time winner of the Loebner Contest (the holy grail for chatbots) http://www.loebner.net/ AIML – Artificial Intelligence Markup Language http://www.aimlbots.com/ PandoraBots http://www.pandorabots.com

9. Embodied Conversational Agents ‘Theory’ behind ALICE -pattern matching -Zipf distribution -Iterative

10. Freudbot 1 Why Freud? Initial plan of deployment The famous personality application –Emile http://www.hud.ac.uk/hhs/research/emile/emileframeset.htm http://www.hud.ac.uk/hhs/research/emile/emileframeset.htm –Shakespeare http://www.pandorabots.com/pandora/talk?botid=c6937cfb3e354738 http://www.pandorabots.com/pandora/talk?botid=c6937cfb3e354738 –Hans Christian Anderson http://www.niceproject.com/about/ http://www.niceproject.com/about/ –John Lennon

11. Freudbot 1 Developing the AIML Narrative structure Test chatters How much ALICE?

12. Research Questions Is it worth it? Is ‘chattiness’ related to the subjective evaluation of chat experience? Are there individual difference variables that are related to measures of chat performance/experience? Freudbot 1

13. Online Recruitment – restricted to psychology students – Incentive (1/30 chance at $300) Random assignment to bot type Controlled Chat –automatically directed to questionnaire after 10 mins of chat Freudbot 1: Methodology

15. nPercent GenderMen1218% Women5582% Age Distribution18-2269% 23-271522% 28-321116% 33-37710% 38-421522% 42+1319% Student StatusFull-time2740% Part-time3552% Non-student58% Self-rated academic Below avg00% abilityAverage1319% Above avg3958% Excellent1522% Freudbot 1: Participants (N=67)

16. Is it worth it? self-report data* Mean Useful 2.2 Recommend2.4 Overall2.4 Enjoyable 2.6 Engaging2.7 Memorable2.8 Expansion3.4 * 5 point scale Would you chat again? Yes No (n=30) (n=35) 2.7 1.8 3.4 1.6 3.2 1.8 3.4 1.9 3.4 2.1 3.6 2.2 4.1 2.8

17. Best Features Interactivity16 Able to ask questions with answers16 Learning about Freud & theories13 Simplicity/ease of use5 Entertaining/humorous5 Thought provoking5 No good features5 Technological features of Freudbot4 Potential to Freudbot4 Alternative learning style3 Novelty/uniqueness of Freudbot3 Tricking Freudbot2 Unpredictable2 Worst Features Repetition33 Unable to answer questions23 Conversation did not flow12 Limited knowledge base10 User needed prior knowledge 3 User was uncertain about what to do 3 Not an effective learning tool3 Conversation was too short1 No sound1 Is it worth it?

18. MeanRange Number of Exchanges31.05-82 Chat logs Mean Proportion of on-task responses by participant*.60 questions.37 comments*.23 * correlated with a composite measure of self rated chat experience Proportion of repetitions by Freudbot.25 Proportion of non-sensical by Freudbot.39

19. FreudAlice JustFreud n=35n=32 Useful2.22.3 Recommend2.52.4 Overall2.52.4 Enjoyable2.7 2.6 Memorable3.02.7 Engaging2.82.7 Expansion3.33.5 Chattiness? # of Exchanges 32.2 29.7 On task Response*.56.64 * -significant difference btw groups

20. Individual difference variables? demographic –Gender –Age –Student status* –Self-rated academic ability computer experience & self-rated skill academic background –# of university courses –# of distance ed courses* –# of psychology courses –Rated importance of Freud*

21. Individual difference variables? attitudes towards technology and education –Positive aspects of on-line activities –Independent Learner –negative aspects of on-line activities*

22. Is it worth it? –worth another look Is ‘chattiness’ related to the subjective evaluation of chat experience? –‘Chattiness’ is not the right level –Nass and Reeves (1998) Are there individual difference variables that are related to measures of chat performance/experience? –some relations that make sense and others that don’t Freudbot 1 Summary

23. Research Goals 1. Improve Performance Fix repetition problem Topic tags More content 2. Replication 3. Instructional Set 4. Future Development Freudbot 2

24. online recruitment, incentive, & controlled chat identical to Freudbot 1 random assignment to instructional set similar questionnaire with additional questions on applications and improvements Freudbot 2:Methodology http://psych.athabascau.ca/html/Freudbot/test.html

25. nPercent GenderMen1018% Women4582% Age Distribution18-22713% 23-271731% 28-32713% 33-371120% 38-42611% 42+713% Student StatusFull-time2647% Part-time2851% Non-student12% Self-rated academic 0-5004% ability50-6524% 66-791120% 80-893055% 90+1018% Participants (N=55)

26. Improvement? self-report data (5 point scale) Freudbot 1 Freudbot 2 Useful**2.23.0 Recommend**2.42.9 Overall**2.43.0 Enjoyable 2.63.0 Engaging**2.73.1 Memorable2.83.1 Expansion**3.44.1 ** - statisically significant Would you chat again? Yes No (n=37) (n=18) 3.3 2.4 3.4 1.7 3.4 2.2 3.3 2.3 3.5 2.2 3.6 2.1 4.4 3.3

27. Improvement? MeanRange Number of Exchanges28.43-115 Chat logs Mean Proportion of on-task responses by participant*.90 questions.36 comments.48 * correlated with a composite measure of self rated chat experience Proportion of appropriate responses by Freud.60

28. Replication? Demographic –Gender* –Age –Student status* –Self-rated academic ability computer experience academic background –# of university courses –# of distance ed courses –# of psychology courses –Rated importance of Freud*

29. Replication? attitudes towards technology and education –Positive aspects of on-line activities –Independent Learner –negative aspects of on-line activities*

30. Instructional Set? Brief Set Elaborate Set n=27n=28 Useful3.12.9 Recommend2.82.9 Overall2.93.1 Enjoyable2.9 3.0 Memorable3.23.0 Engaging3.03.3 Expansion3.94.2 # of Exchanges 25.3 31.3 On task Response.90.90

31. Future Development? Freudbot Improvements Mean* Chat behaviour4.2 Audio Response3.1 Voice Recognition2.6 Synchronization2.5 Animation/movment2.3 * 5-point scale Other Applications Mean* Practice quizbot4.1 Famous personality4.1 Course content3.4 Chatroom3.3 Course Admin3.2

32. 1. Improvement - yes, but clearly room for more 2. Replication - some 3. Instructional Set - no effects 4. Development Freudbot 2: Summary

33. Future Direction Haptek Freud –Animacy/agency hypothesis http://psych.athabascau.ca/html/Freudbot/haptek.html Piagetbot (Support from MCR) –learning outcomes Skinnerbot (Lyle Grant) Coursebot Quizbot

34. Questions?