Conversational Agents and Learning Outcomes: An Experimental Investigation Bob Heller Mike Proctor Athabasca University Research supported by Mission Critical Research Grant - Athabasca University EDMEDIA, Vancouver, 2007

Conversational agents as pedagogical agents AutoTutor - conversational intelligent tutor system (see Graesser, Wiemer-Hastings, Wiemer-Hastings, Kreuz, & Tutoring Research Group 1999) Embodied Conversational Agents - primacy of conversation (Cassells et al) Pedagogical Agent research juncture of HCI and educational psychology (Hadwin, Winne, & Nesbit, 2005)

The Persona Effect Pedagogical agents have the potential to increase the bandwidth of communication between students and computers and the computer’s ability to engage and motivate students (Johnson, Rickel, and Lester, 2000) Reeves & Nass The Media Equation

Evidence for Persona Effect

Objectives of the Present Study Learning Outcome Historical Figure replication

Method Participants Materials Procedure

Logical Interface

Chat Interface

Interface Evaluation Text InterfaceChat Interface n=28 n=31 M MSD t How enjoyable was this activity? * How engaging was this activity How easy was this activity? Would you recommend this activity to others? * Overall, how would you rate this activity? * How useful is this interface for learning information about Jean Piaget? ** How useful is this interface for remembering information about Jean Piaget? * Table 1: Interface evaluation ratings based on a 5-point scale where higher numbers reflect positive experiences. Note: * - p <.05 ; ** - p <.01

TextChatp Pretest46.9%38.3%ns Content Exposure83.8%56.3%<.001 Exposed Content 67.4%56.8%<.05 Non-exposed Content 26.8%33.0%ns

Freudbot Piagetbot n=53 n=31 p How enjoyable was this activity? ns How engaging was this activity? ns Would you recommend this activity to others? ns Overall, how would you rate this activity? ns How useful is this interface for learning information about Freud/Piaget? ns Table 2: Ratings based on a 5-point scale where higher numbers reflect positive experiences. Note: All t’s were non-significant Agent Evaluation

Discussion - the text-based interface was rated more positively and associated with better learning outcomes Limitations - orienting task biased towards text-based interface - degree of animation

Freudbot AIML Narrative format Speech Act theory

Freudbot findings

Conversational agents as pedagogical agents