Towards lifelike Computer Interfaces that learn Towards lifelike Computer Interfaces that learn & Motion Capture System 2007.4.18 James Lee Electronic Visualization Lab University of Illinois at Chicago

Electronic Visualization Laboratory, University of Illinois at Chicago Agenda Project Introduction Overview Alex & AlexDSS Simplified Model 3 year goals Motion Capture System Generals Workflow Example Electronic Visualization Laboratory, University of Illinois at Chicago

Electronic Visualization Laboratory, University of Illinois at Chicago Overview 3 yrs project: NSF sponsor Collaboration UIC: EVL + Communication (Steve Jones) Intelligent System Laboratory, the University of Central Florida in Orlando (Avelino Gonzalez, Ronald DeMara) Objective Develops and evaluates lifelike, natural computer interfaces as portals to intelligent programs in the context of Decision Support System (DSS) An avatar-based interface supporting spoken natural language combined with gestural expressions Extends a current NSF-sponsored DSS-based project Electronic Visualization Laboratory, University of Illinois at Chicago

Electronic Visualization Laboratory, University of Illinois at Chicago Alex Who is Alex? Dr. Alex Schwarzkopf Long standing NSF I/UCRC program manager Regarded as the expert Going to retire Model of avatar project! * NSF I/UCRC: Industry & University Cooperative Research Center Electronic Visualization Laboratory, University of Illinois at Chicago

Electronic Visualization Laboratory, University of Illinois at Chicago AlexDSS What is AlexDSS? NSF sponsored project Capture, preserve, and allow the reuse of the knowledge and expertise Dr. Schwarzkopf has acquired in his position as program manager Help answer commonly asked questions about I/UCRC program AskAlex Web Interface Electronic Visualization Laboratory, University of Illinois at Chicago

Electronic Visualization Laboratory, University of Illinois at Chicago Simplified Model Knowledge Processing Text based Knowledge Speech Recognition Voice to Text Expression Synthesizer Text to Speech (TTS) Lip Synch & Emotion Answer Animate User: Questions Idle Animation Feed Dr. Schwarzkopf Virtual Avatar Electronic Visualization Laboratory, University of Illinois at Chicago

Electronic Visualization Laboratory, University of Illinois at Chicago 3 year goals Year 1 Year 2 Year 3 EVL Initial Design of Avatar System Framework Integrate sub-modules - SR, AlexDSS High-res lifesize avatar Motion capture - basic gestures Improve voice synthesis - Schwarzkopf’s voice Non-verbal communicate - eye tracking Advanced recognition - camera based user tracking Enhanced eye tracking UCF evaluate/select SR sys Spoken Interface SR optimization - real-time execution Evaluate heterogeneous input equipment for SR Initial hybrid Speech/ pointing Interface Future platform study - distributed architecture Electronic Visualization Laboratory, University of Illinois at Chicago

Electronic Visualization Laboratory, University of Illinois at Chicago Motion Capture System Performance Capture! “Actors don’t do motion, they do emotion”, James Cameron Image that you are shooting movie scene in general. Candidate System VICON MX40 8 cameras (2352 x 1728 resolution) Motion Analysis ? Plan Where: BATCAVE area (maybe current c-wall space) When: Summer 07 (full utilization from Fall 07) Electronic Visualization Laboratory, University of Illinois at Chicago

Electronic Visualization Laboratory, University of Illinois at Chicago H/W & S/W Electronic Visualization Laboratory, University of Illinois at Chicago

Electronic Visualization Laboratory, University of Illinois at Chicago Capture Procedure Phase 1 Preparation Phase 2 Capturing Phase 3 Post-processing Better be prepared Any kind of description - Storyboard - Synopsis - Sketches and etc Cast good performer Perform capture Highly depends on system H/W, S/W Motion retarget Check adaptation Refine motion Fine-tune rigging If necessary, re-capture scene Electronic Visualization Laboratory, University of Illinois at Chicago

Electronic Visualization Laboratory, University of Illinois at Chicago Example Polar Express Sony Pictures Imageworks Inc. 2005 4 actors at the same time with Full body and facial track 80 markers for body, 152 markers for face(232 per actor) 16 cams for body, 56 cams for face At peak, 928 markers @ 60 fps Electronic Visualization Laboratory, University of Illinois at Chicago

Electronic Visualization Laboratory, University of Illinois at Chicago Example King Kong Universal Pictures 2005 Electronic Visualization Laboratory, University of Illinois at Chicago

Electronic Visualization Laboratory, University of Illinois at Chicago Workshop Possible workshop for prospective users EVLers (CS + Art students) Related Classes students Video Game Design, Computer Graphics Virtual Reality and etc Topics System introduction and operation brief Performance Capture Pipeline When? Late summer or beginning of fall 07 Electronic Visualization Laboratory, University of Illinois at Chicago

Electronic Visualization Laboratory, University of Illinois at Chicago Thank you! Questions? Electronic Visualization Laboratory, University of Illinois at Chicago