A FACEREADER- DRIVEN 3D EXPRESSIVE AVATAR Crystal Butler | Amsterdam 2013

2 Computer Vision OBJECT DETECTION AND CATEGORIZATION, FOR EXAMPLE: FACIAL RECOGNITION 2

3 Machine Learning PATTERN RECOGNITION BASED ON A SET OF DATA, FOR EXAMPLE SPAM FILTERING 3 Support Vector Machine

4 Motion Capture USED FOR CREATING LIFELIKE MOVEMENT IN VIDEO GAMES AND ANIMATION 4

5 3D Modeling A COMPUTER GRAPHICS METHOD FOR REPRESENTING 3-DIMENSIONAL OBJECTS, USED IN GAMES AND ANIMATION 5 Polygon Mesh

6 UV Map A 2-DIMENSIONAL TEXTURE MAP APPLIED TO A 3D MODELED SURFACE 6 Maya

7 Depth Mapping WITH THE KINECT, INFRARED POINTS ARE PROJECTED AND READ BY THE SENSOR TO RENDER A DEPTH MODEL OF THE ENVIRONMENT 7 Kinect

Technology and Features Avatar Creation Process

Goal: Use FaceReader’s Action Unit, Head and Eye Poses, and Mask Data to drive a 3D Avatar that Mirrors User Expressions in Real timeFaceReader’s

10 faceshift for Modeling AUTOMATED MODELING USING THE KINECT 10

11 Maya for Motion and Texture MANUAL ADJUSTMENTS AND ADDITIONS OF BLENDSHAPES 11

12 About us Read the Action Unit values from the live FaceReader feed via a comma-delimited text file Update the blendshapes in Maya using those AUs Get the head angle values from the FaceReader text file and apply them to the neck joint MEL Commands Scripting in Maya 12 Get the face mask texture from the live FaceReader feed and apply it to the UV map on the model Grab a point color from the current mask texture and apply it to the shading node for the head Set an animation key frame so the capture can be played back Check for a radio button change indicating that the current facial texture should be fixed

13 The Avatar Now FACEREADER VS FACESHIFT SMACKDOWN 13

14 Hindering Video DO EXPRESSIONS OF DISGUST REDUCE FEELINGS OF HAPPINESS? 14

WHY AVATARS?

16 Controlled by a computer program, for example Siri or CleverbotCleverbot Agents Avatars vs The Difference 16 Controlled by a human being, for example video game characters

17 This group from Northeastern University, USA, is working on medical applications including: Relational Agents Group Some Current Research AVATARS AND AGENTS Affective Social Computing Lab Rachael Jack Institute for Creative Technologies 17 Christine Lisetti, Florida International University, develops FACS-based virtual counselors to provide mental healthcare: “ On-Demand VIrtual Counselor (ODVIC) In this project, we design and implement the prototype of On- Demand VIrtual Counselor (ODVIC) intelligent virtual characters who can provide people access to effective behavior change interventions and help them find and cultivate motivation to change unhealthy lifestyles (e.g. excessive alcohol use, overeating). An empathic Embodied Conversational Agent (ECA) delivers the intervention. The health dialog is directed by a computational model of Motivational Interviewing, a novel effective face-to-face patient- centered counseling style which respects an individual’s pace toward behavior change. “ Dr. Jack’s work at the University of Glasgow uses avatars generated by a FACS-based facial grammar to create expressions and test recognition across cultures: This University of Southern California group investigates a variety of uses for virtual humans, focusing on education and training:

18 The Beta PRIOR TO APPLYING TEXTURES OR ADJUSTING BLENDSHAPES (AND TO SHOW OFF VOICE ACTIVATION!) 18

?