2 USC Institute for Creative Technologies Eric Forbell, Nicolai Kalisch, Kelly Christoffersen, Glenn Storm, Kevin Feeley, Fabrizio Morbini SimCoach – SimCoach – Authoring

2 SimCoach as a Platform  SimCoach is a technology platform that enables rapidly developed virtual humans hosted in a web environment –Enables more ubiquitous and scalable virtual humans and content –Builds upon maturing virtual human technologies –Built upon open web standards, promoting adoption –Supports true conversational agents that take initiative – Suitable for a variety of complex domains –Enables interactivity with a vast set of existing and future web resources – Direct embedding of YouTube, Web articles, Website links, other online media

3 Building authorable virtual humans Multimodal Perception & Understanding Human User Animation Natural Language Generation Speech Generation Speech Recognition Nonverbal Behavior Generation Renderer Natural Language Understanding Agent Framework Challenges Virtual Humans have complex data-driven components Tools are typically not unified and unwieldy for non-technical authors Balance between simplicity and expressivity Different applications impose different requirements Training is more focused, but requires assessment Support systems like SimCoach face more uncertainty with users

4 SimCoach Authoring Approach Preliminary Goals Build a unified tool to create complete virtual characters Author, test and iterate – all in the same environment Minimize barriers for getting started Tool is fully web-based, nothing to install or configure Use available technologies, invent only when necessary Empower the author, restrict only when necessary Construct a solid architectural foundation to support domain-specific authoring applications

5 SimCoach Architecture – What needs authoring? Human User Textual Language Understanding Web client Animation Fusion Speech Audio Non-verbal Behavior generation Framework Dialogue Management Animation Video Action Realization Multimedia Resources Speech Analysis Classifier model Dialogue policy Information state Speech Audio Behavior Rules Animation Library Resources Utterances 3D Character

6 SimCoach Authoring Tools Early design iterations

7 Authoring Process  Create project –Select 3D character model from library question.confidentialityconsentToPTSDQuestionnaire == true Question.what-is-PTSD  Build interaction logic –Compose interaction through sub-dialogue scripts activated by state variables and user actions  Train language understanding –For all specified user dialogue actions, provide exemplar text and variants  Refine character actions –Edit character utterances –Import speech audio, render animations

8 Enabling technology for web-hosted characters Smartbody FaceFX NVBG encoder Maya Maya Maya Maya Maya Maya Can parallelize here! Automated Character Animation Workflow

9 Kraken Character Animation Production System Create Speech Audio Analyze Speech Audio Create Nonverbal behavior Fuse and Animate Render Frames Encode Video createAudio analyzeAudio createBml createBvh encodeMovie createAudio analyzeAudio createBml createBvh buildAndRend er encodeMovie New ticket Processor Ticket finish Processor Authoring Tools

10 TOOLS DEMO PRE-ALPHA TECH PREVIEW

11 Future Work  Build tutorials and in-tool support for new authors  Improve overall usability –Some changes may be very invasive, but necessary  Simplify creation of common dialogue behaviors –Introduce behavior templates  Improve author control and preview of nonverbal behavior –In process on DCAPS  Enhance testing framework to help debug  Provide support for other simulation environments –Standalone, non-web simulations

12 Backups

13

14

15

16

17 TOOLS DEMO PRE-ALPHA TECH PREVIEW