Technische Universität München Face Model Fitting based on Machine Learning from Multi-band Images of Facial Components Institute for Informatics Technische Universität München Germany Matthias Wimmer Christoph Mayer Freek Stulp Bernd Radig

2/22 Face Model Fitting based on Machine Learning from Multi-band Images of Facial Components Technische Universität München – Matthias Wimmer Outline of this Presentation Infer semantic information Model fitting with learned objective functions Compute multi-band image representation image facial expression, gaze, identity, gender, age,… Part 2: Part 1: facial component correctly fitted face model

3/22 Face Model Fitting based on Machine Learning from Multi-band Images of Facial Components Technische Universität München – Matthias Wimmer Part 1: Compute Multi-band Image Representing Facial Components

4/22 Face Model Fitting based on Machine Learning from Multi-band Images of Facial Components Technische Universität München – Matthias Wimmer Motivation and Related Work  Descriptive feature representation of input image  Input for the subsequent process of model fitting  Quick computation  Similar approaches:  Stegman et al. (IVC2003) Stegman et al. Image and Vision Computing (2003)

5/22 Face Model Fitting based on Machine Learning from Multi-band Images of Facial Components Technische Universität München – Matthias Wimmer Our Idea  Multi-band image contains the location of facial components  Use classifier for this task.  Provide a multitude of features  Classifier decides which ones are relevant (→ quick)  Consider pixel features only (→ quick)  Pre-compute image characteristics and adjust pixel features (→ accurate) skin lips teeth brows pupils

6/22 Face Model Fitting based on Machine Learning from Multi-band Images of Facial Components Technische Universität München – Matthias Wimmer Prerequisites  Image data base (from Web; 500 images)  Face Locator: e.g. Viola and Jones  Computes rectangular regions around human faces  Manual annotations

7/22 Face Model Fitting based on Machine Learning from Multi-band Images of Facial Components Technische Universität München – Matthias Wimmer Probability Matrices  Indicate the probability of the pixels to denote a certain facial component.  Relative to the face rectangle  Learned offline skin brows pupils lower lip teeth

8/22 Face Model Fitting based on Machine Learning from Multi-band Images of Facial Components Technische Universität München – Matthias Wimmer Image Characteristics  Describe characteristics of the entire image  Computed by applying probability mask to face rectangle color  Distribution of color of all facial components  Gaussian distribution of color (Mean, covariance matrix) space  Distribution of pixel locations of all facial components  Gaussian distribution of locations (Mean, covariance matrix) example imagespatial distribution of skin colorskin color distribution

9/22 Face Model Fitting based on Machine Learning from Multi-band Images of Facial Components Technische Universität München – Matthias Wimmer Multitude of Pixel Features color Static pixel features  Color (RGB, NRGB, HSV, YCbCr) 16 features Adjusted pixel features  Color relative to mean color (Euclidean, Mahalanobis) ~ 90 features space  Coordinates (Cartesian, Polar)  Coordinates relative to mean location of facial components (Euclidean, Mahalanobis)

10/22 Face Model Fitting based on Machine Learning from Multi-band Images of Facial Components Technische Universität München – Matthias Wimmer Evaluation: Train various Classifiers  4 Classifiers for each facial component  C1: static feature only  C2: adjusted color features only  C3: adjusted location features only  C4: all features

11/22 Face Model Fitting based on Machine Learning from Multi-band Images of Facial Components Technische Universität München – Matthias Wimmer Classifiers for Lips and Teeth C1C2C3C4 77.3%94.3%90.4%97.7% C1C2C3C4 74.3%66.2%87.9%95.0%

12/22 Face Model Fitting based on Machine Learning from Multi-band Images of Facial Components Technische Universität München – Matthias Wimmer Part 2: Model Fitting with Learned Objective Functions

13/22 Face Model Fitting based on Machine Learning from Multi-band Images of Facial Components Technische Universität München – Matthias Wimmer Model-based Image Interpretation  The model The model contains a parameter vector that represents the model’s configuration.  The objective function Calculates a value that indicates how accurately a parameterized model matches an image.  The fitting algorithm Searches for the model parameters that describe the image best, i.e. it minimizes the objective function.

14/22 Face Model Fitting based on Machine Learning from Multi-band Images of Facial Components Technische Universität München – Matthias Wimmer Local Objective Functions

15/22 Face Model Fitting based on Machine Learning from Multi-band Images of Facial Components Technische Universität München – Matthias Wimmer Ideal Objective Functions P1:Correctness property: Global minimum corresponds to the best fit. P2:Uni-modality property: The objective function has no local extrema. ¬ P1 P1 ¬P2 P2  Don’t exist for real-world images  Only for annotated images: f n ( I, x ) = | c n – x |

16/22 Face Model Fitting based on Machine Learning from Multi-band Images of Facial Components Technische Universität München – Matthias Wimmer Learning the Objective Function x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x  Ideal objective function generates training data  Machine Learning technique generates calculation rules ideal objective function training data learned objective function

17/22 Face Model Fitting based on Machine Learning from Multi-band Images of Facial Components Technische Universität München – Matthias Wimmer Benefits of the Machine Learning Approach  Accurate and robust calculation rules  Locally customized calculation rules  Generalization from many images  Simple job for the designer  Critical decisions are automated  No domain-dependent knowledge required  No loops

18/22 Face Model Fitting based on Machine Learning from Multi-band Images of Facial Components Technische Universität München – Matthias Wimmer Evaluation 1: Displacing the Correct Model statistics-based objective function ideal objective function learned objective function

19/22 Face Model Fitting based on Machine Learning from Multi-band Images of Facial Components Technische Universität München – Matthias Wimmer Evaluation 2: Selected Features contour point 116

20/22 Face Model Fitting based on Machine Learning from Multi-band Images of Facial Components Technische Universität München – Matthias Wimmer Conclusion  Crucial decisions within Computer Vision algorithms  Don’t solve by trial and error → Learn from training data  Example 1: Learned classifiers for facial components  Example 2: Learned objective functions

21/22 Face Model Fitting based on Machine Learning from Multi-band Images of Facial Components Technische Universität München – Matthias Wimmer Outlook  More features for learning objective functions  Higher number of features  Other kinds of features: SIFT, LBP, …  Learn with better classifiers  Relevance Vector Machines  Boosted regressors  Training images: render faces with AAM  Exact ground truth (no manual work required)  Many images  Learn global objective function  Learn rules to directly update model parameter

22/22 Face Model Fitting based on Machine Learning from Multi-band Images of Facial Components Technische Universität München – Matthias Wimmer Thank you! Online-Demonstration: