Sunee Holland University of South Australia School of Computer and Information Science Supervisor: Dr G Stewart Von Itzstein

 What is Facial Expression Synthesis?  Background  Motivation  Methodology › Research Questions  Implementation › Why the Source SDK? › System Architecture › Script Design › Voice Recording › Source SDK – Face Poser › Source SDK – Hammer  User Study  Results  Conclusion

 Simulation of human facial communication into graphical form  The goal is to increase realism and enhance the user’s experience  Can extend facial expression synthesis techniques to show deception leakage in virtual faces

 Techniques for facial expression synthesis › Motion capture › Muscle based actuation › Deformations › Morphing or blending  Toolkits › Source SDK (Face Poser), Xface, BEAT, Face Toolkit, Expression

 Embodied Conversational Agents (ECAs) › Realistic, virtual avatars › Gestures › Facial expressions › Speech  Psychology › Deceptive expressions in humans › Paul Ekman’s universal emotions › Micro expressions › Duchenne de Boulogne

 The aim of this research is to evaluate the means of communication between human and computer › Specifically, deception in virtual agents › Using micro expressions  Literature covers how facial expression synthesis is performed  We want to focus on what these systems can be used for

 Question the feasibility of deceptive facial expression synthesis  Develop a deceptive facial expression synthesis technique  Evaluate this technique through a user study

 Is it feasible to create a facial expression synthesis technique that portrays deception? › Is the user able to detect and recognise deception leakage on a computer generated model? › What effect does the synthesised deception leakage of an animated 3D character have on the user's experience ?

 What are the challenges in implementing a technique for synthesising deception leakage in computer-generated characters? › What existing software is available for synthesising facial deception leakage and what is the optimal choice for portraying deception? › How can deception be implemented in facial expression synthesis with the tools currently available?

 Using the Source SDK  Design the scenarios used in the experiment  Generate audio dialogue to create choreographies in Face Poser  Use those choreographies to make maps in Hammer  Map runs through the Source Engine in Half Life 2

 Other toolkits evaluated: › Xface, rFace, Expression  Need high level of detail so that subtlety can be accurately portrayed

 Need to create a script for the scenarios to be included in the experiment  As we want to evaluate the expressions only, must remove other confounding factors  Contextual bias could be a confounding factor › Bad: “I can’t wait to see your parents. It will be great” › Good: “I can’t wait to see my parents. It will be great”  Need to express emotions equally › Anger, disgust, fear, happiness, sadness and surprise, Duchenne (genuine) smile and Pan American (non- genuine) smile  Scenarios need to be realistic

 Choosing voice actors › One male and one female  They were given a script and asked to voice the dialogue › Recorded using computer software › Encoded into a format Face Poser works with › WAV 16-bit PCM  The speech was required to be said in a neutral tone as to avoid bias induced in the voice

 SDK tool that produces choreographed sequences  Audio files are imported › Phoneme extraction/lip synching  Choreographies are created using a timeline of events › Contains expressions and audio  Result is exported into Valve Choreography Data (VCD) file

 SDK tool to create, edit, and export maps for Source Engine games  Build the physical geometry  Place entities in the map › Player spawn location, triggers, map changes, choreographies, characters  Place an event on a generic character that links to a choreography file  Compile and run through Half Life 2

 In the process of finalising the experiment  Participants › Ideal population: over 30 participants › Students, researchers and staff  Training phase › Some people will have a predisposition to recognising deception › Reduce this confounding factor by placing all participants on a similar experience level

 1. Evaluating the recognition of expressions › How well does the technique express emotion? › User is shown an expression or scenario and asked whether they thought it was genuine or deceptive, and/or what emotion they thought they were perceiving

 2. Evaluating subjective user experience › What level of quality is the interaction between computer and user? › Participants will be asked to fill out a feedback form › Subjective feedback wrt how easily they could identify deception and emotions

 Predict that there will be difficulty in accurately recognising deception › Similar with humans  However, it should show a recognition rate that is slightly greater than chance › This is all that is required to validate the research

 The main contribution of this research is the evaluation of facial expression synthesis portraying deception  The user study will answer the question regarding feasibility through recognition rate and subjective user experience  Question regarding challenges is answered in the implementation through selecting a pre existing solution and describing how the deception was implemented

 More complex facial expressions  More ways of portraying deception other than facially › Body leakage is important when detecting deception  Possible applications in gaming to improve the immersive experience

 Want a way to randomise the choreography files for every participant in the experiment  Solution: Custom software that inputs files from a location, randomises the ordering, then outputs in the Half Life 2 scenes directory  Written in C#