Human Emotion Synthesis David Oziem, Lisa Gralewski, Neill Campbell, Colin Dalton, David Gibson, Barry Thomas University of Bristol, Motion Ripper, 3CR Research

Synthesising Facial Emotions – University of Bristol – 3CR Research Project Group Motion Ripper Project –Methods of motion capture. –Re-using captured motion signatures. –Synthesising new or extend motion sequences. –Tools to aid animation. Collaboration between University of Bristol CS, Matrix Media & Granada.

Synthesising Facial Emotions – University of Bristol – 3CR Research Introduction What is an emotion? Ekman outlined 6 different basic emotions. –joy, disgust, surprise, fear, anger and sadness. Emotional states relate to ones expression and movement. Synthesising video footage of an actress expressing different emotions.

Synthesising Facial Emotions – University of Bristol – 3CR Research

Video Textures Video textures or temporal textures are textures with motion. (Szummer’96) Schodl’00, reordered frames from the original to produce loops or continuous sequences. –Doesn’t produce new footage. Campbell’01, Fitzgibbon’01, Reissell’01, used Autoregressive process (ARP) to synthesis frames. Examples of Video Textures

Synthesising Facial Emotions – University of Bristol – 3CR Research Autoregressive Process Statistical model Calculating the model involves working out the parameter vector (a 1 …a n ) and w. n is known as the order of the sequence. y(t) = – a 1 y(t – 1) – a 2 y(t – 2) – … – a n y(t – n) + w.ε Parameter vector (a1,…,an)Noise Current value at time t

Synthesising Facial Emotions – University of Bristol – 3CR Research Autoregressive Process Statistical model Increasing dimensionality of y drastically increases the complexity in calculating (a 1 …a n ). y(t) = – a 1 y(t – 1) – a 2 y(t – 2) – … – a n y(t – n) + w.ε

Synthesising Facial Emotions – University of Bristol – 3CR Research Autoregressive Process PCA analysis of Sad footage in 2D Secondary mode Primary mode Principal Components Analysis is used to reduce number of dimensions in the original sequence.

Synthesising Facial Emotions – University of Bristol – 3CR Research Autoregressive Process PCA analysis of Sad footage in 2DGenerated sequence using an ARP Secondary mode Primary mode Non-Gaussian Distribution is incorrectly modelled by an ARP.

Synthesising Facial Emotions – University of Bristol – 3CR Research Face Modelling Campbell’01, synthesised a talking head. Cootes and Talyor’00, combined appearance model. –Isolates shape and texture. Requires labelled frames. –Must label important features on the face. Labelled points

Synthesising Facial Emotions – University of Bristol – 3CR Research Combined Appearance Shape space Hand Labelled video footage provides a point set which represents the shape space of the clip.

Synthesising Facial Emotions – University of Bristol – 3CR Research Combined Appearance Shape space Texture space Warping each frame into a standard pose, creates the texture space. The standard pose is the mean position of the points.

Synthesising Facial Emotions – University of Bristol – 3CR Research Combined Appearance Shape space Texture space Combined space Joining the shape and texture space and then re-analysing using PCA produces the combined space.

Synthesising Facial Emotions – University of Bristol – 3CR Research Combined Appearance Shape space Texture space Combined space Reconstruction of the original sequence from the combined space. Combined space

Synthesising Facial Emotions – University of Bristol – 3CR Research Secondary mode Primary mode Combined Appearance Combined Appearance sequence Original sequence in 2D Secondary mode Primary mode Change in distribution after applying The combined appearance technique

Synthesising Facial Emotions – University of Bristol – 3CR Research Secondary mode Primary mode Combined Appearance Generated Sequence Original sequence Secondary mode Primary mode ARP model Visually the generated plot appears to have been generated using the same stochastic process as the original.

Synthesising Facial Emotions – University of Bristol – 3CR Research Copying and ARP Combine the benefits of copying with ARP –New motion signatures. –Handles non-Gaussian distributions.

Synthesising Facial Emotions – University of Bristol – 3CR Research Copying and ARP Original input Reduced input PCA Important to reduce the complexity of the search process. Need around 30 to 40 dimensions in this example.

Synthesising Facial Emotions – University of Bristol – 3CR Research Copying and ARP Original input Reduced input Segmented input PCA Reduced segments PCA Temporal segments of between 15 to 30 frames. Need to reduce each segment to be able to train ARP’s.

Synthesising Facial Emotions – University of Bristol – 3CR Research Copying and ARP Original input Reduced input Segmented input Reduced segments PCA ARP Synthesised segments Many of the learned models are unstable % are usable.

Synthesising Facial Emotions – University of Bristol – 3CR Research Copying and ARP Original input Reduced input Segmented input Reduced segments PCA ARP Synthesised segments Segment selection Outputted Sequence

Synthesising Facial Emotions – University of Bristol – 3CR Research Example First mode Time t End of generated sequence. Possible segments. Compared section

Synthesising Facial Emotions – University of Bristol – 3CR Research First mode Time t Example Closest 3 segments are chosen.

Synthesising Facial Emotions – University of Bristol – 3CR Research First mode Time t Example The segment to be copied is randomly selected from the closest 3.

Synthesising Facial Emotions – University of Bristol – 3CR Research First mode Time t Example Segments are blended together using a small overlap and averaging the overlapping pixels.

Synthesising Facial Emotions – University of Bristol – 3CR Research Secondary mode Primary mode Secondary mode Primary mode Copying & ARP model PCA analysis of Sad footage in 2D Generated sequence Copying and ARP Potentially infinitely long. Includes new novel motions.

Synthesising Facial Emotions – University of Bristol – 3CR Research Results (Angry) Source FootageCopying with ARPCombined Appearance ARP Combined appearance produces higher resolution frames. Better motion from the copying and ARP approach

Synthesising Facial Emotions – University of Bristol – 3CR Research Results (Sad) Source FootageCopying with ARPCombined Appearance ARP Similar results as with the angry footage –Copied approach is less blurred due to the reduced variance.

Synthesising Facial Emotions – University of Bristol – 3CR Research Comparison Results - Combined appearance - Segment copying Simple objective comparison. –Randomly selected temporal segments.

Synthesising Facial Emotions – University of Bristol – 3CR Research Comparison Perceptually is it better to have good motion or higher resolution.

Synthesising Facial Emotions – University of Bristol – 3CR Research Combined appearanceSegment Copying with ARP

Synthesising Facial Emotions – University of Bristol – 3CR Research Other potential uses Self Organising Map Uses combined appearance –as each ARP model provides a minimal representation of the given emotion. Can navigate between emotions to create new interstates. Angry Sad Happy

Synthesising Facial Emotions – University of Bristol – 3CR Research Conclusions Both methods can produce synthesised clips of a given emotion. Combined appearance produces higher definition frames. Copying and ARPs generates more natural movements.

Synthesising Facial Emotions – University of Bristol – 3CR Research Questions