2. The Entertainment Industry Meets Simulation: The Challenge of the Holodeck Bill Swartout Institute for Creative Technologies University of Southern California

3. The ICT is intended to serve as a nexus between the simulation and entertainment communities

4. Biz Tech Show Institute For Creative Technologies

5. How can we best simulate reality? Two approaches –Military simulation –Entertainment industry

6. Simulating Reality: Two Approaches Military Simulation Literal: try to include as much reality in simulation as possible Homogeneous Entertainment Industry Goal-directed: create effect of real experience by selectively focusing on elements (real and unreal) Hybrid Approach: combine disparate elements

7. Movie clip

8. To the Holodeck and beyond…

9. Mission Rehearsal Exercise Project: New Operations

10. Mission Rehearsal Exercise Project: ICT VR Theater

11. What we don’t want… Movie clip

12. New Research Areas Speech Recognition Natural Language Understanding Dialogue Management Natural Language Generation Speech Synthesis Gesture Generation Perception Personality Variation Sound Spatialization Interactive Story Structure

13. Movie clip

14. Integration Raises New Challenges: Natural Language Project Leaders: Ed Hovy (ISI), Kevin Knight (ISI), David Traum (ISI), Shri Narayanan (IMSC) Speech recognition: Noisy environment Multimodal: Face To Face (speech+gesture), Radio Multiple Interactors (Sgt, medic, squads,…) –Messages tailored for multiple addressees/overhearers Emotion

15. Most speech synthesis sounds unnatural Even the best have limited expression (AT&T NextGen) –No command voice Use hybrid approach: –Multiple synthesizers with different “voices” –Command voice Text to Speech Synthesis “Squad Leaders listen up!” Challenge: Speech Synthesis Project Leaders: Lewis Johnson (ISI), Shri Narayanan (IMSC) Use pre-recorded library for highly emotional speech

16. Challenge: Modeling Emotions Project Leaders: Jon Gratch (ICT), Stacy Marsella (ISI)

17. Movie clip

18. Challenge: Modeling Emotions Project Leaders: Jon Gratch (ICT), Stacy Marsella (ISI) Problem addressed: –Virtual humans need emotions to be believable & realistic Approach: –Characters have beliefs, goals, utilities –Construct plans –Events evaluated against plans –If goals thwarted (or enabled) characters react

19. Emotional Reaction Movie clip

20. Modeling Emotions: Issues Integrating gesture, natural language & voice to project realistic behavior Managing unintended interpretations Hollywood vs. Psychology Movie emotions quite different from natural emotions Recognizing human emotions

21. Challenge: Story is Critical More than an event list Successful stories: –Juxtapose contrasting elements –Surprise (plot twists) –Dilemmas –Things go wrong –Build to climax (and ebb)

22. Structuring Stories for Intelligent Agents: Issues Traditional story telling tricks require linear structure Immersed participant needs freewill Need “bumpers” –Possibility: director agent controls environment & characters to limit choice

23. Reasons for Optimism Hybrid approach –Use all the tools in the toolbox Entertainment industry as source of knowledge and techniques Story context constrains possibilities Synergy from graphics, sound & emotional engagement –The mind is a willing collaborator