1. Towards lifelike Computer Interfaces that learn
Towards lifelike Computer Interfaces that learn & Motion Capture System
James Lee
Electronic Visualization Lab
University of Illinois at Chicago

2. Electronic Visualization Laboratory, University of Illinois at Chicago
Agenda
Project Introduction
Overview
Alex & AlexDSS
Simplified Model
3 year goals
Motion Capture System
Generals
Workflow
Example
Electronic Visualization Laboratory, University of Illinois at Chicago

3. Electronic Visualization Laboratory, University of Illinois at Chicago
Overview
3 yrs project: NSF sponsor
Collaboration
UIC: EVL + Communication (Steve Jones)
Intelligent System Laboratory, the University of Central Florida in Orlando (Avelino Gonzalez, Ronald DeMara)
Objective
Develops and evaluates lifelike, natural computer interfaces as portals to intelligent programs in the context of Decision Support System (DSS)
An avatar-based interface supporting spoken natural language combined with gestural expressions
Extends a current NSF-sponsored DSS-based project
Electronic Visualization Laboratory, University of Illinois at Chicago

4. Electronic Visualization Laboratory, University of Illinois at Chicago
Alex
Who is Alex?
Dr. Alex Schwarzkopf
Long standing NSF I/UCRC program manager
Regarded as the expert
Going to retire
Model of avatar project!
* NSF I/UCRC: Industry & University Cooperative Research Center
Electronic Visualization Laboratory, University of Illinois at Chicago

5. Electronic Visualization Laboratory, University of Illinois at Chicago
AlexDSS
What is AlexDSS?
NSF sponsored project
Capture, preserve, and allow the reuse of the knowledge and expertise Dr. Schwarzkopf has acquired in his position as program manager
Help answer commonly asked questions about I/UCRC program
AskAlex Web Interface
Electronic Visualization Laboratory, University of Illinois at Chicago

6. Electronic Visualization Laboratory, University of Illinois at Chicago
Simplified Model
Knowledge
Processing
Text based
Knowledge
Speech
Recognition
Voice to Text
Expression Synthesizer
Text to Speech (TTS)
Lip Synch & Emotion
Answer
Animate
User: Questions
Idle Animation Feed
Dr. Schwarzkopf
Virtual Avatar
Electronic Visualization Laboratory, University of Illinois at Chicago

7. Electronic Visualization Laboratory, University of Illinois at Chicago
3 year goals
Year 1
Year 2
Year 3
EVL
Initial Design of Avatar
System Framework
Integrate sub-modules
- SR, AlexDSS
High-res lifesize avatar
Motion capture
- basic gestures
Improve voice synthesis
- Schwarzkopf’s voice
Non-verbal communicate
- eye tracking
Advanced recognition
- camera based user
tracking
Enhanced eye tracking
UCF
evaluate/select SR sys
Spoken Interface
SR optimization
- real-time execution
Evaluate heterogeneous
input equipment for SR
Initial hybrid Speech/
pointing Interface
Future platform study
- distributed architecture
Electronic Visualization Laboratory, University of Illinois at Chicago

8. Electronic Visualization Laboratory, University of Illinois at Chicago
Motion Capture System
Performance Capture!
“Actors don’t do motion, they do emotion”, James Cameron
Image that you are shooting movie scene in general.
Candidate System
VICON MX40 8 cameras (2352 x 1728 resolution)
Motion Analysis ?
Plan
Where: BATCAVE area (maybe current c-wall space)
When: Summer 07 (full utilization from Fall 07)
Electronic Visualization Laboratory, University of Illinois at Chicago

9. Electronic Visualization Laboratory, University of Illinois at Chicago
H/W & S/W
Electronic Visualization Laboratory, University of Illinois at Chicago

10. Electronic Visualization Laboratory, University of Illinois at Chicago
Capture Procedure
Phase 1
Preparation
Phase 2
Capturing
Phase 3
Post-processing
Better be prepared
Any kind of description
- Storyboard
- Synopsis
- Sketches and etc
Cast good performer
Perform capture
Highly depends on
system H/W, S/W
Motion retarget
Check adaptation
Refine motion
Fine-tune rigging
If necessary, re-capture scene
Electronic Visualization Laboratory, University of Illinois at Chicago

11. Electronic Visualization Laboratory, University of Illinois at Chicago
Example
Polar Express
Sony Pictures Imageworks Inc. 2005
4 actors at the same time with Full body and facial track
80 markers for body, 152 markers for face(232 per actor)
16 cams for body, 56 cams for face
At peak, fps
Electronic Visualization Laboratory, University of Illinois at Chicago

12. Electronic Visualization Laboratory, University of Illinois at Chicago
Example
King Kong
Universal Pictures 2005
Electronic Visualization Laboratory, University of Illinois at Chicago

13. Electronic Visualization Laboratory, University of Illinois at Chicago
Workshop
Possible workshop for prospective users
EVLers (CS + Art students)
Related Classes students
Video Game Design, Computer Graphics
Virtual Reality and etc
Topics
System introduction and operation brief
Performance Capture Pipeline
When?
Late summer or beginning of fall 07
Electronic Visualization Laboratory, University of Illinois at Chicago

14. Electronic Visualization Laboratory, University of Illinois at Chicago
Thank you!
Questions?
Electronic Visualization Laboratory, University of Illinois at Chicago