Prediction of Non-Linear Aging Trajectories of Faces K. Scherbaum, M. Sunkel, V. Blanz and H.-P. Seidel [ 2007/5/9, Eurographics 2007, Prague ]

Motivation / Goal automated growth-prediction system applications photofit-pictures of missing children automated animation, art Kristina Scherbaum scherbaum@mpi-inf.mpg.de

Age Progression – Optimal Case 9 years 10 years 11 years child 1 child 2 challenges: depends on individual face depends on age (curved trajectory) no longitudinal study child 3 face space Kristina Scherbaum scherbaum@mpi-inf.mpg.de

Real Case – Support Vector Regression only 1 sample per person no longitudinal study find isosurfaces and gradients 9 years 10 years 11 years 11 years 9 years 10 years challenges: depends on individual face depends on age (curved trajectory) no longitudinal study Runge-Kutta Integration face space Kristina Scherbaum scherbaum@mpi-inf.mpg.de

Main Assumption - Curved Trajectories growing faces transform along curved trajectories use machine learning non-linear Support Vector Regression integration of local age-gradient challenges: depends on individual face depends on age (curved trajectory) no longitudinal study Kristina Scherbaum scherbaum@mpi-inf.mpg.de

Challenges learn change over time of individual faces non-linear dependency on time, curved trajectory learn how the change depends on individual face non-linear dependency in face space sparse dataset, no longitudinal study challenges: depends on individual face depends on age (curved trajectory) no longitudinal study Kristina Scherbaum scherbaum@mpi-inf.mpg.de

3D Morphable Facemodel System is based on a Morphable 3D Facemodel [Blanz,Vetter‘99] Built from 200 3D-face-scans of adults Kristina Scherbaum scherbaum@mpi-inf.mpg.de

3D Morphable Facemodel vector space of faces vectors with point-to-point correspondence Shape linear combinations of faces Texture Kristina Scherbaum scherbaum@mpi-inf.mpg.de

Representation of Faces - Face Spaces PCA to reduce dimensionality (yields coefficients) 3 different representations: vector representation linear combinations pca space (reduced complexity), variation In the morphable model, faces are represented as vectors for the shape “S” and for the texture “T” each linear combination of different faces is a new realistic face in order to reduce complexity a PCA was applied on the shape and texture vectors this defines an orthogonal set of basis vectors si and ti with the average shape and texture we can define shape and texture coefficients c_is and c_it Kristina Scherbaum scherbaum@mpi-inf.mpg.de

Extended Morphable Model Extension by … plus ~238 facemodels of teenagers 3 simultaneous laser scans per face Correspondence by … top-down approach fitting Morphable Model to new 3D faces merging original data and best fit Evtl erweitern: 2-3 folien Evtl. auch Simultanfit erklaeren Beispiele der Teenagerdatenbank zeigen (Diplomarbeit S 58 und S 59) Zoom, zeigen dass Korrepondenz vorhanden ist (evtl morph) und beleuchtung normiert wurde evtl noch die Texturreko zeigen. Kristina Scherbaum scherbaum@mpi-inf.mpg.de

Fitting the Morphable Model to 3D Scans no optical flow because scans are often incomplete best fit of the morphable model merged result 3D laser scans Kristina Scherbaum scherbaum@mpi-inf.mpg.de

Texture Extraction from Pictures 3 pictures per face under inconsistent lighting conditions view dependent mapping result with normalised lighting 3D reco extracted textures pictures Kristina Scherbaum scherbaum@mpi-inf.mpg.de

Age Progression Algorithm 1 learn function that maps any face x to a scalar age y to learn this function we use … non-linear Support-Vector-Regression on training sets of l pairs f: Face  Age R^n  R maps any face x to a scalar age value let a face x consist either of shape coefficients or texture coefficients to reduce the complexity we do not use all principal components (k=20,40,80) yi denotes the age of eac h example face i Kristina Scherbaum scherbaum@mpi-inf.mpg.de

Fitting a Regression Curve 2 for a given set of samples find f(x) such that all samples are within an e-tube preselect e and tradeoff between smoothness and errors of outliers y e e x f: Face  Age gross darstellen evtl bilder dazu R^n  R discuss linear methods at the end x Linear: f(x) = wx + b Non-linear: f is sum of Gaussian RBF kernels K(x-xi) Kristina Scherbaum scherbaum@mpi-inf.mpg.de

Non-Linear SVM Regression 2 Gaussian RBF (Radial Basis Function) as kernel we applied grid search using cross validation to optimize parameters such as g (Kernelwidth) i and b are determined by SVM training using LIBSVM for e-Support Vector Regression a_i and be are real numbered values determined by the SVM training set of 20, 40 and 80 PCs are used in cross validation we split the dataset in 11 different random ways into 90% training and 10% test faces Kristina Scherbaum scherbaum@mpi-inf.mpg.de

3 Local Aging Isosurfaces are defined in PCA space Gradient gives shortest path to next isosurface Along the gradient … many facial changes due to aging almost no other changes (known technique, Blanz et al. 99) Thus: Compute growth along the gradient! Kristina Scherbaum scherbaum@mpi-inf.mpg.de

Gradient Example - Facial Attributes 3 gender manipulation male female original Kristina Scherbaum scherbaum@mpi-inf.mpg.de

Growth Simulation: New Approach 3 growth curve with given face x0 at time t currently we compute the local gradient and walk along this gradient instead we should compute the curved trajectory Kristina Scherbaum scherbaum@mpi-inf.mpg.de

Runge Kutta Integration 4 Solve differential equation … to compute curved trajectories integrate the differential equation using Runge-Kutta algorithm perform small steps for all x and t the gradient of f describes the direction of minimal change in x to achieve a given change in t so that the characteristic features of the face are retained in the best way possible Runge Kutta (4th order): performs small steps along the gradient of f the gradient of f can be computed from the RBF regression function Kristina Scherbaum scherbaum@mpi-inf.mpg.de

Visualized Aging Trajectories 4 Kristina Scherbaum scherbaum@mpi-inf.mpg.de

4 Reducing Complexity growth leads to overall change of facial size we did not train on all principle components speedup of SVM training we experimented with 20, 40 or 80 PCs Justification … growth leads to overall change of facial size significant changes are represented by the first PCs [ large variance ] facial growth should happen in the first PCs Kristina Scherbaum scherbaum@mpi-inf.mpg.de

Growth Example growth simulation for both, shape and texture 12 14 16 18 20 years 22 24 26 28 30 years Kristina Scherbaum scherbaum@mpi-inf.mpg.de

More Examples 10 years 3D laser scans, original age 12 13 12 10 14 look at the bottom line – different individuals not always the same result / type characteristic features remain 20 years 30 years Kristina Scherbaum scherbaum@mpi-inf.mpg.de

Rendering the Result into Images [EG’04] Background, Haircut Pose, Light Face Composed Result 3D reconstruction and aging Kristina Scherbaum scherbaum@mpi-inf.mpg.de

Photofit Picture Example Input at the age of 11 Possible appearances at the age of 17 Kristina Scherbaum scherbaum@mpi-inf.mpg.de

Aging in Images - Example Picture (1999) Different prediction renderings 3D reconstruction and aging Ground truth pictures (2005) Kristina Scherbaum scherbaum@mpi-inf.mpg.de

3D reconstruction aging (extrapolated) Extrapolated Example GROUND TRUTH picture 2 years old prediction, 14 years 3D reconstruction aging (extrapolated) extreme example database did not contain babies or toddlers Kristina Scherbaum scherbaum@mpi-inf.mpg.de

Linear vs. Non-Linear Linear age progression Disadvantages … perform linear regression (yields a function) [ straight-forward least squares fit ] transform faces also along the gradient Disadvantages … the gradient is constant [ linear function ] each face moves along the same straight trajectory pro und contra linear zum vergleich nochmal linear am ende – SVR schon erklaert. neue fit-funktion zum vergleich Kristina Scherbaum scherbaum@mpi-inf.mpg.de

Linear vs. Non-Linear comparison of age estimation error (in months) mean squared training and generalization errors non-linear (RBF) 32.68 18.12 linear 66.14 60.05 non-linear (RBF) 38.90 29.35 linear 67.66 62.87 (here for 20 PCs) non-linear SVM regression behaves superior! generalization indicates: no overfitting Kristina Scherbaum scherbaum@mpi-inf.mpg.de

Remember the Challenges Are growth trajectories curved? Mean angle between start- and target-tangent 10.3º 30.0º  the trajectories are curved, not linear Have different faces distinct trajectories? Mean angle of trajectories of different faces 15.7º 33.5º  the trajectories are different challenges: depends on individual face depends on age (curved trajectory) no longitudinal study Kristina Scherbaum scherbaum@mpi-inf.mpg.de

Conclusions Results … aging involves non-linear components trajectories are distinct for different individuals linear systems are a reasonable approximation technique works without longitudinal data But … more data would be helpful longitudinal data would allow for exact evaluation Kristina Scherbaum scherbaum@mpi-inf.mpg.de

Thank you for your attention! MOVIE Kristina Scherbaum scherbaum@mpi-inf.mpg.de

Transforming Faces Along Trajectories 5 given face x0 with estimated start age test = f(x0) simulate trajectory z along a time period t = ttarget - test compute a face vector for the target age ttarget z fuer from und textur getrennt! if t0 and test differ compensate for error (shift face) ensures that face does not look too young or too old Kristina Scherbaum scherbaum@mpi-inf.mpg.de

Observations mean angles as non-linearity measures between … trajectories of different faces linear and RBF gradients start- and target-tanget of a face aging trajectories are curved! they tend to be linear for more PCs Kristina Scherbaum scherbaum@mpi-inf.mpg.de

Evaluation Prediction is not truly evaluated … Possible improvements … no 3D ground-truth data available exact evaluation not possible we reconstruct a 3D model from pictures as ground truth but the reconstruction is always the best guess only Possible improvements … extend the database of teenagers, re-scan faces today user studies (prediction from pictures, comparison) considering side effects (parents look, nutrition, smoking …) Kristina Scherbaum scherbaum@mpi-inf.mpg.de

Piecewise Linear Approach given input face and age averages find start and target age range compute start and target average face subtract and add scaled personality Kristina Scherbaum scherbaum@mpi-inf.mpg.de

Simple Solution - Piecewise linear simple and fast solution easy to calculate sufficient results but … no smooth transitions sometimes individuals are recognizable more data needed SVR is a more fundamental machine learning approach Kristina Scherbaum scherbaum@mpi-inf.mpg.de

Representation of Faces - Face Spaces arbitrary faces by linear combinations of examples PCA to reduce dimensionality (yields coefficients) 3 different representations: vector representation linear combinations pca space (reduced complexity), variation In the morphable model, faces are represented as vectors for the shape “S” and for the texture “T” each linear combination of different faces is a new realistic face in order to reduce complexity a PCA was applied on the shape and texture vectors this defines an orthogonal set of basis vectors si and ti with the average shape and texture we can define shape and texture coefficients c_is and c_it Kristina Scherbaum scherbaum@mpi-inf.mpg.de

4 Aging Trajectories Main Idea … compute aging trajectories z(t) locally along gradient of the aging function f(x) and going through a start vector or face x0: for all x and t the gradient of f describes the direction of minimal change in x to achieve a given change in t so that the characteristic features of the face are retained in the best way possible Runge Kutta (4th order): performs small steps along the gradient of f the gradient of f can be computed from the RBF regression function Kristina Scherbaum scherbaum@mpi-inf.mpg.de

Aging Information extracted from the database of 200 adult face scans and new database of 238 face scans of teenagers teenager overview Kristina Scherbaum scherbaum@mpi-inf.mpg.de

Kristina Scherbaum scherbaum@mpi-inf.mpg.de