Learning the Appearance of Faces: A Unifying Approach for the Analysis and Synthesis of Images. Thomas Vetter Germany University of Freiburg

Computer Vision & Computer Graphics Computer Graphics can help to solve Computer Vision! | G (p) - I | 2 = min Parameters G ( image ) Parameters V ision ( image ) parameters image G raphics ( parameters )

Analysis by Synthesis 3D World Image Analysis Synthesis Image Model Image Description model parameter

Synthesis of Faces Input Image Modeler Result Database Face Analyzer 3D Head Morphable Face Model

Approach: Example based modeling of faces 2D Image 3D Face Models = w 1 * + w 2 * + w 3 * + w 4 * +...

Cylindrical Coordinates red(h,  ) green(h,  ) blue(h,  ) red(h,  ) green(h,  ) blue(h,  )  h radius(h,  ) h 

Morphing 3D Faces 3D Blend 3D Morph 1 __ =

Correspondence: A two step process! Correspondence between 1. two examples ( Optical Flow like algorithms). 2. many examples ( Morphable Model ) Reference Example 2 nd Example

= a 1 * + a 2 * + a 3 * + a 4 * +... b 1 * + b 2 * + b 3 * + b 4 * +... Vector space of 3D faces. A Morphable Model can generate new faces.

Manipulation of Faces Modeler

Modelling in Face Space Caricatur Original Average

Modelling the Appearance of Faces A face is represented as a point in face space. Which directions code for specific attributes ?

Learning from Labeled Example Faces Fitting a (linear) regression function

Facial Attributes WeightWeight OriginalOriginal Subjective Attractiveness

Transfer of Facial Expressions = Smile - - Originals: + Smile = Novel Face:

Facial Expressions OriginalOriginal

3D Shape from Images Face Analyzer 3D Head Input Image

Matching a Morphable 3D-Face-Model = R Optimization problem! a 1 * + a 2 * + a 3 * + a 4 * +.. b 1 * + b 2 * + b 3 * + b 4 * +..

Error Function Image difference Plausible parameters Minimize

Optimization Strategies Stochastic Gradient Decent Difference Decomposition

Future Challenges Which Object Classes are linear ? How to built them automatically?