1. Prediction of Non-Linear Aging Trajectories of Faces
K. Scherbaum, M. Sunkel, V. Blanz and H.-P. Seidel
[ 2007/5/9, Eurographics 2007, Prague ]

2. Motivation / Goal automated growth-prediction system applications
photofit-pictures of missing children
automated animation, art
Kristina Scherbaum

3. Age Progression – Optimal Case
9 years
10 years
11 years
child 1
child 2
challenges:
depends on individual face
depends on age (curved trajectory)
no longitudinal study
child 3
face space
Kristina Scherbaum

4. Real Case – Support Vector Regression
only 1 sample per person
no longitudinal study
find isosurfaces
and gradients
9 years
10 years
11 years
11 years
9 years
10 years
challenges:
depends on individual face
depends on age (curved trajectory)
no longitudinal study
Runge-Kutta Integration
face space
Kristina Scherbaum

5. Main Assumption - Curved Trajectories
growing faces transform along curved trajectories
use machine learning
non-linear Support Vector Regression
integration of local age-gradient
challenges:
depends on individual face
depends on age (curved trajectory)
no longitudinal study
Kristina Scherbaum

6. Challenges
learn change over time of individual faces non-linear dependency on time, curved trajectory
learn how the change depends on individual face non-linear dependency in face space
sparse dataset, no longitudinal study
challenges:
depends on individual face
depends on age (curved trajectory)
no longitudinal study
Kristina Scherbaum

7. 3D Morphable Facemodel
System is based on a Morphable 3D Facemodel [Blanz,Vetter‘99]
Built from 200 3D-face-scans of adults
Kristina Scherbaum

8. 3D Morphable Facemodel vector space of faces
vectors with point-to-point correspondence
Shape
linear combinations of faces
Texture
Kristina Scherbaum

9. Representation of Faces - Face Spaces
PCA to reduce dimensionality (yields coefficients)
3 different representations:
vector representation
linear combinations
pca space (reduced complexity), variation
In the morphable model, faces are represented as vectors for the shape “S” and for the texture “T”
each linear combination of different faces is a new realistic face
in order to reduce complexity a PCA was applied on the shape and texture vectors
this defines an orthogonal set of basis vectors si and ti
with the average shape and texture we can define shape and texture coefficients c_is and c_it
Kristina Scherbaum

10. Extended Morphable Model
Extension by …
plus ~238 facemodels of teenagers
3 simultaneous laser scans per face
Correspondence by …
top-down approach
fitting Morphable Model to new 3D faces
merging original data and best fit
Evtl erweitern:
2-3 folien
Evtl. auch Simultanfit erklaeren
Beispiele der Teenagerdatenbank zeigen (Diplomarbeit S 58 und S 59) Zoom, zeigen dass Korrepondenz vorhanden ist (evtl morph) und beleuchtung normiert wurde
evtl noch die Texturreko zeigen.
Kristina Scherbaum

11. Fitting the Morphable Model to 3D Scans
no optical flow because scans are often incomplete
best fit of the morphable model
merged result
3D laser scans
Kristina Scherbaum

12. Texture Extraction from Pictures
3 pictures per face under inconsistent lighting conditions
view dependent mapping
result with normalised lighting
3D reco
extracted textures
pictures
Kristina Scherbaum

13. Age Progression Algorithm1
learn function
that maps any face x to a scalar age y
to learn this function we use …
non-linear Support-Vector-Regression
on training sets of l pairs
f: Face  Age R^n  R maps any face x to a scalar age value
let a face x consist either of shape coefficients or texture coefficients
to reduce the complexity we do not use all principal components (k=20,40,80)
yi denotes the age of eac h example face i
Kristina Scherbaum

14. Fitting a Regression Curve2
for a given set of samples find f(x)
such that all samples are within an e-tube
preselect e
and tradeoff between smoothness and errors of outliers y e e x
f: Face  Age gross darstellen evtl bilder dazu
R^n  R
discuss linear methods at the end x
Linear: f(x) = wx + b
Non-linear: f is sum of Gaussian RBF kernels K(x-xi)
Kristina Scherbaum

15. Non-Linear SVM Regression2
Gaussian RBF (Radial Basis Function) as kernel
we applied grid search using cross validation
to optimize parameters such as g (Kernelwidth)
i and b are determined by SVM training
using LIBSVM for e-Support Vector Regression
a_i and be are real numbered values determined by the SVM training
set of 20, 40 and 80 PCs are used
in cross validation we split the dataset in 11 different random ways into 90% training and 10% test faces
Kristina Scherbaum

16. 3 Local Aging Isosurfaces are defined in PCA space
Gradient gives shortest path to next isosurface
Along the gradient …
many facial changes due to aging
almost no other changes (known technique, Blanz et al. 99)
Thus: Compute growth along the gradient!
Kristina Scherbaum

17. Gradient Example - Facial Attributes3
gender manipulation
male
female
original
Kristina Scherbaum

18. Growth Simulation: New Approach3
growth curve with given face x0 at time t
currently we compute the local gradient
and walk along this gradient
instead we should compute the curved trajectory
Kristina Scherbaum

19. Runge Kutta Integration4
Solve differential equation …
to compute curved trajectories
integrate the differential equation
using Runge-Kutta algorithm
perform small steps
for all x and t the gradient of f describes the direction of minimal change in x to achieve a given change in t so that the characteristic features of the face are retained in the best way possible
Runge Kutta (4th order):
performs small steps
along the gradient of f
the gradient of f can be computed from the RBF regression function
Kristina Scherbaum

20. Visualized Aging Trajectories4
Kristina Scherbaum

21. 4 Reducing Complexity growth leads to overall change of facial size
we did not train on all principle components
speedup of SVM training
we experimented with 20, 40 or 80 PCs
Justification …
growth leads to overall change of facial size
significant changes are represented by the first PCs [ large variance ]
facial growth should happen in the first PCs
Kristina Scherbaum

22. Growth Example growth simulation for both, shape and texture 12 14 1618 20
years 22 24 26 28 30
years
Kristina Scherbaum

23. More Examples 10 years 3D laser scans, original age 12 13 12 10 14
look at the bottom line – different individuals not always the same result / type
characteristic features remain
20 years
30 years
Kristina Scherbaum

24. Rendering the Result into Images [EG’04]
Background, Haircut
Pose, Light
Face
Composed Result
3D reconstruction and aging
Kristina Scherbaum

25. Photofit Picture Example
Input at the age of 11
Possible appearances at the age of 17
Kristina Scherbaum

26. Aging in Images - Example
Picture (1999)
Different prediction renderings
3D reconstruction and aging
Ground truth pictures (2005)
Kristina Scherbaum

27. 3D reconstruction aging (extrapolated)
Extrapolated Example
GROUND TRUTH
picture 2 years old
prediction, 14 years
3D reconstruction aging (extrapolated)
extreme example
database did not contain babies or toddlers
Kristina Scherbaum

28. Linear vs. Non-Linear Linear age progression Disadvantages …
perform linear regression (yields a function) [ straight-forward least squares fit ]
transform faces also along the gradient
Disadvantages …
the gradient is constant [ linear function ]
each face moves along the same straight trajectory
pro und contra linear
zum vergleich nochmal linear am ende – SVR schon erklaert. neue fit-funktion zum vergleich
Kristina Scherbaum

29. Linear vs. Non-Linear comparison of age estimation error (in months)
mean squared training and generalization errors
non-linear (RBF)
32.68
18.12
linear
66.14
60.05
non-linear (RBF)
38.90
29.35
linear
67.66
62.87
(here for 20 PCs)
non-linear SVM regression behaves superior!
generalization indicates: no overfitting
Kristina Scherbaum

30. Remember the Challenges
Are growth trajectories curved? Mean angle between start- and target-tangent
10.3º
30.0º
 the trajectories are curved, not linear
Have different faces distinct trajectories? Mean angle of trajectories of different faces
15.7º
33.5º
 the trajectories are different
challenges:
depends on individual face
depends on age (curved trajectory)
no longitudinal study
Kristina Scherbaum

31. Conclusions Results … aging involves non-linear components
trajectories are distinct for different individuals
linear systems are a reasonable approximation
technique works without longitudinal data
But …
more data would be helpful
longitudinal data would allow for exact evaluation
Kristina Scherbaum

32. Thank you for your attention!
MOVIE
Kristina Scherbaum

33. Transforming Faces Along Trajectories5
given face x0 with estimated start age test = f(x0)
simulate trajectory z along a time period t = ttarget - test
compute a face vector for the target age ttarget
z fuer from und textur getrennt!
if t0 and test differ compensate for error (shift face)
ensures that face does not look too young or too old
Kristina Scherbaum

34. Observations mean angles as non-linearity measures between …
trajectories of different faces
linear and RBF gradients
start- and target-tanget of a face
aging trajectories are curved!
they tend to be linear for more PCs
Kristina Scherbaum

35. Evaluation Prediction is not truly evaluated … Possible improvements …
no 3D ground-truth data available
exact evaluation not possible
we reconstruct a 3D model from pictures as ground truth
but the reconstruction is always the best guess only
Possible improvements …
extend the database of teenagers, re-scan faces today
user studies (prediction from pictures, comparison)
considering side effects (parents look, nutrition, smoking …)
Kristina Scherbaum

36. Piecewise Linear Approach
given input face and age averages
find start and target age range
compute start and target average face
subtract and add scaled personality
Kristina Scherbaum

37. Simple Solution - Piecewise linear
simple and fast solution
easy to calculate
sufficient results
but …
no smooth transitions
sometimes individuals are recognizable
more data needed
SVR is a more fundamental machine learning approach
Kristina Scherbaum

38. Representation of Faces - Face Spaces
arbitrary faces by linear combinations of examples
PCA to reduce dimensionality (yields coefficients)
3 different representations:
vector representation
linear combinations
pca space (reduced complexity), variation
In the morphable model, faces are represented as vectors for the shape “S” and for the texture “T”
each linear combination of different faces is a new realistic face
in order to reduce complexity a PCA was applied on the shape and texture vectors
this defines an orthogonal set of basis vectors si and ti
with the average shape and texture we can define shape and texture coefficients c_is and c_it
Kristina Scherbaum

39. 4 Aging Trajectories Main Idea … compute aging trajectories z(t)
locally along gradient of the aging function f(x)
and going through a start vector or face x0:
for all x and t the gradient of f describes the direction of minimal change in x to achieve a given change in t so that the characteristic features of the face are retained in the best way possible
Runge Kutta (4th order):
performs small steps
along the gradient of f
the gradient of f can be computed from the RBF regression function
Kristina Scherbaum

40. Aging Information extracted from the database of 200 adult face scans
and new database of 238 face scans of teenagers
teenager overview
Kristina Scherbaum

41. Kristina Scherbaum scherbaum@mpi-inf.mpg.de