Embodied Conversational Agents: A Case Study of Freudbot Bob Heller, PhD Athabasca University November 3, 2004

Mike Proctor – AIML programmer Dean Mah – Web implementation Billy Cheung – Graphics, test chatter Lisa Jewell – Chat log analysis, content developer, test chatter Julianna Charchun – Chat log analysis Jude Onuh – AIML programmer Acknowledgements

Embodied Conversational Agents Definitions Embodiment in Conversational Interfaces: REA (Cassel et al., 1999) Embodied Conversational Agents (Cassel, Sullivan, Prevost, & Churchill, 2000) –FMTB model Vos (2002) offers 5 features of ECA –Human like appearance –Body used for communication purposes –Natural communication protocols –Multimodality –Social role

Embodied Conversational Agents Anthropomorphic Agents Animated Interface Agents Animated Pedagogical agents Pedagogical Agent Persona Intelligent Tutoring Systems -AutoTutor (Graesser et al) Chatterbots or Chatbots - Weizenbaum’s (1966) Eliza

Why? -primacy of conversation -Constructivist theory -The Media Equation -Persona effect -cognitive load Embodied Conversational Agents

Richard Wallace and A.L.I.C.E. Artificial Linguistic Internet Computer Entity 3 time winner of the Loebner Contest (the holy grail for chatbots) AIML – Artificial Intelligence Markup Language PandoraBots

Embodied Conversational Agents ‘Theory’ behind ALICE -pattern matching -Zipf distribution -Iterative

Freudbot 1 Why Freud? Initial plan of deployment The famous personality application –Emile –Shakespeare –Hans Christian Anderson –John Lennon

Freudbot 1 Developing the AIML Narrative structure Test chatters How much ALICE?

Research Questions Is it worth it? Is ‘chattiness’ related to the subjective evaluation of chat experience? Are there individual difference variables that are related to measures of chat performance/experience? Freudbot 1

Online Recruitment – restricted to psychology students – Incentive (1/30 chance at $300) Random assignment to bot type Controlled Chat –automatically directed to questionnaire after 10 mins of chat Freudbot 1: Methodology

nPercent GenderMen1218% Women5582% Age Distribution % % % % % % Student StatusFull-time2740% Part-time3552% Non-student58% Self-rated academic Below avg00% abilityAverage1319% Above avg3958% Excellent1522% Freudbot 1: Participants (N=67)

Is it worth it? self-report data* Mean Useful 2.2 Recommend2.4 Overall2.4 Enjoyable 2.6 Engaging2.7 Memorable2.8 Expansion3.4 * 5 point scale Would you chat again? Yes No (n=30) (n=35)

Best Features Interactivity16 Able to ask questions with answers16 Learning about Freud & theories13 Simplicity/ease of use5 Entertaining/humorous5 Thought provoking5 No good features5 Technological features of Freudbot4 Potential to Freudbot4 Alternative learning style3 Novelty/uniqueness of Freudbot3 Tricking Freudbot2 Unpredictable2 Worst Features Repetition33 Unable to answer questions23 Conversation did not flow12 Limited knowledge base10 User needed prior knowledge 3 User was uncertain about what to do 3 Not an effective learning tool3 Conversation was too short1 No sound1 Is it worth it?

MeanRange Number of Exchanges Chat logs Mean Proportion of on-task responses by participant*.60 questions.37 comments*.23 * correlated with a composite measure of self rated chat experience Proportion of repetitions by Freudbot.25 Proportion of non-sensical by Freudbot.39

FreudAlice JustFreud n=35n=32 Useful Recommend Overall Enjoyable Memorable Engaging Expansion Chattiness? # of Exchanges On task Response* * -significant difference btw groups

Individual difference variables? demographic –Gender –Age –Student status* –Self-rated academic ability computer experience & self-rated skill academic background –# of university courses –# of distance ed courses* –# of psychology courses –Rated importance of Freud*

Individual difference variables? attitudes towards technology and education –Positive aspects of on-line activities –Independent Learner –negative aspects of on-line activities*

Is it worth it? –worth another look Is ‘chattiness’ related to the subjective evaluation of chat experience? –‘Chattiness’ is not the right level –Nass and Reeves (1998) Are there individual difference variables that are related to measures of chat performance/experience? –some relations that make sense and others that don’t Freudbot 1 Summary

Research Goals 1. Improve Performance Fix repetition problem Topic tags More content 2. Replication 3. Instructional Set 4. Future Development Freudbot 2

online recruitment, incentive, & controlled chat identical to Freudbot 1 random assignment to instructional set similar questionnaire with additional questions on applications and improvements Freudbot 2:Methodology

nPercent GenderMen1018% Women4582% Age Distribution % % % % % % Student StatusFull-time2647% Part-time2851% Non-student12% Self-rated academic % ability % % % % Participants (N=55)

Improvement? self-report data (5 point scale) Freudbot 1 Freudbot 2 Useful** Recommend** Overall** Enjoyable Engaging** Memorable Expansion** ** - statisically significant Would you chat again? Yes No (n=37) (n=18)

Improvement? MeanRange Number of Exchanges Chat logs Mean Proportion of on-task responses by participant*.90 questions.36 comments.48 * correlated with a composite measure of self rated chat experience Proportion of appropriate responses by Freud.60

Replication? Demographic –Gender* –Age –Student status* –Self-rated academic ability computer experience academic background –# of university courses –# of distance ed courses –# of psychology courses –Rated importance of Freud*

Replication? attitudes towards technology and education –Positive aspects of on-line activities –Independent Learner –negative aspects of on-line activities*

Instructional Set? Brief Set Elaborate Set n=27n=28 Useful Recommend Overall Enjoyable Memorable Engaging Expansion # of Exchanges On task Response.90.90

Future Development? Freudbot Improvements Mean* Chat behaviour4.2 Audio Response3.1 Voice Recognition2.6 Synchronization2.5 Animation/movment2.3 * 5-point scale Other Applications Mean* Practice quizbot4.1 Famous personality4.1 Course content3.4 Chatroom3.3 Course Admin3.2

1. Improvement - yes, but clearly room for more 2. Replication - some 3. Instructional Set - no effects 4. Development Freudbot 2: Summary

Future Direction Haptek Freud –Animacy/agency hypothesis Piagetbot (Support from MCR) –learning outcomes Skinnerbot (Lyle Grant) Coursebot Quizbot

Questions?