1. 7/17/ :54 AM
Conveying User Facial Expressions into Virtual World E-Learning Applications
Mike Procter
GSRDD 2014
© 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

2. Goal
to investigate usability of inexpensive consumer-grade devices for affect detection
Part of MScIS thesis
“A Multi-agent Framework to support User-Aware Conversational Agents in an E-learning Environment”
Supervisors Fuhua (Oscar) Lin, Bob Heller

3. Background Conversational Agents
Freudbot and Piagetbot
A role-playing actor agent that simulates speaking with Sigmund Freud or Jean Piaget about their life, theories, and colleagues. (Heller & Procter, 2011)
Uses conversational rules to switch or progress through topics.

4. Background 3D Virtual Worlds
Second Life, OpenSim, Open Wonderland
Embodied Conversational Agent (ECA) (Cassel 2001)
Animated Pedagogical Agent (APA) (Lester, 2000)
AU Island (Second Life and OpenSim), Montclair U Theorists Project, Sim-on-a-Stick

6. Background Brain Control Interface (BCI)
Consumer devices (mostly for gamers)
Emotiv EPOC headset
Question: Can we use the Emotiv device to make Freudbot aware of the user’s emotions?

7. A Prototype System
Facial expression and emotional state detected and analysed by emotion agent on user’s computer.
frown
bored
attentive
Summarised emotion data transmitted to user avatar in virtual world
smile
confused
Emotion data is transmitted from user avatar to Freudbot
frown
attentive
bored
smile
confused

8. Demonstration
AU Science and Technology Showcase at Telus World of Science, May 2014
Adults & children participate using Emotiv headset
Freudbot tells a joke and gauges user response
Main issues: hair, sensors, individual differences

9. Conclusions & Future Work
Proof of concept
To be useful need to
Improve the reliability of the interface (more sophisticated s/w, training)
Incorporate other affective cues relevant to learning process (boredom, interest, frustration, …)
Update e-learning application (e.g. Freudbot) to make use of these cues to improve learning outcomes, engage user
Controlled user trials

10. Questions? References Acknowledgements
Cassell, J. (2001) "Embodied Conversational Agents: Representation and Intelligence in User Interface" AI Magazine, Winter 2001, 22(3):
Johnson. W.L., Rickel, J.W., & Lester, J.C. (2000). Animated pedagogical agents: Face-to-face interaction in interactive learning environments. International Journal of Artificial Intelligence in Education, 11,
Heller, R.B. & Procter, M. (2011). Embedded and embodied intelligence: Virtual actors on virtual stages. In S Graf, F Lin, Kinhsuk, R& R. McGreal (Eds.) Intelligent and Adaptive Learning Systems: Technology Enhanced Support for Learners and Teachers. Athabasca University Press.
Acknowledgements
Graduate Student Research Fund
Alberta Innovates Graduate Student Scholarship