1. Data-driven Methods: Faces
Portrait of
Piotr Gibas
© Joaquin
Rosales
Gomez (2003)
CS194: Image Manipulation & Computational Photography
Alexei Efros, UC Berkeley, Fall 2017

2. The Power of Averaging

3. 8-hour exposure
© Atta Kim

4. Image Composites Multiple Individuals Composite Sir Francis Galton
Composite
[Galton, “Composite Portraits”, Nature, 1878]

5. Average Images in Art Idris Khan Krzysztof Pruszkowski
“Spherical type gasholders” (2004)
Idris Khan
“60 passagers de 2e classe du metro, entre 9h et 11h”   (1985) 
Krzysztof Pruszkowski

6. “100 Special Moments” by Jason Salavon
Why
blurry?

7. Object-Centric Averages by Torralba (2001)
Manual Annotation and Alignment
Average Image
Slide by Jun-Yan Zhu

8. Computing Means Two Requirements: Alignment of objects
Objects must span a subspace
Useful concepts:
Subpopulation means
Deviations from the mean

9. Images as Vectors n = m
n*m

10. Vector Mean: Importance of Alignment= = m
½ ½ = mean image
n*m
n*m

11. How to align faces?

12. Shape Vector =
Provides alignment! 43

13. Appearance Vectors vs. Shape Vectors
Vector of
200*150*3
Dimensions
Requires Annotation
Provides alignment!
200*150 pixels (RGB)
Shape
Vector
You can think about faces in different ways. You can represent this way or that way. These are complements.
Vector of
43*2
Dimensions
43 coordinates (x,y)
Slide by Kevin Karsch

14. Average Face 1. Warp to mean shape 2. Average pixels

15. Objects must span a subspace
(0,1)
(.5,.5)
(1,0)

16. Example
mean
Does not span a subspace

17. Subpopulation means Examples: Average female Average kid
Male vs. female
Happy vs. said
Average Kids
Happy Males
Etc.
Average female
We might be curious what different populations of faces look like.
Average kid
Average happy male
Average male

18. Average Women of the world
You will notice that women from every place is very attractive because the …..

19. Average Men of the world

20. Deviations from the mean-
Image X
Mean X =
DX = X - X

21. Deviations from the meanX
DX = X - X + =
+ 1.7 =

22. Extrapolating faces We can imagine various meaningful directions. Sad
Masculine
Current face
Feminine
Happy
Slide by Kevin Karsch

23. Manipulating faces
How can we make a face look more female/male, young/old, happy/sad, etc.?
Current face
Sub-mean 1
----- Meeting Notes (10/4/11 17:24) -----
Add a matlab thing to show the differences. show how to use PCA in matlab.
Sub-mean 2
Slide by Kevin Karsch

24. Manipulating Facial Appearance through Shape and Color
Duncan A. Rowland and David I. Perrett
St Andrews University
IEEE CG&A, September 1995

25. Face Modeling Compute average faces (color and shape)
Compute deviations between male and female (vector and color differences)

26. Changing gender
Deform shape and/or color of an input face in the direction of “more female”
original shape
color both

27. more same original androgynous more opposite
Enhancing gender
more same original androgynous more opposite

28. Changing age Face becomes “rounder” and “more textured” and “grayer”
original shape
color both

29. Back to the Subspace

30. Linear Subspace: convex combinations
Any new image X can be
obtained as weighted sum
of stored “basis” images.
Our old friend, change of basis!
What are the new coordinates
of X?

31. The Morphable Face Model
The actual structure of a face is captured in the shape vector S = (x1, y1, x2, …, yn)T, containing the (x, y) coordinates of the n vertices of a face, and the appearance (texture) vector T = (R1, G1, B1, R2, …, Gn, Bn)T, containing the color values of the mean-warped face image.
Shape S
Appearance T

32. The Morphable face model
Again, assuming that we have m such vector pairs in full correspondence, we can form new shapes Smodel and new appearances Tmodel as:
If number of basis faces m is large enough to span the face subspace then:
Any new face can be represented as a pair of vectors
(a1, a2, ... , am)T and (b1, b2, ... , bm)T !

33. Issues: How many basis images is enough? Which ones should they be?
What if some variations are more important than others?
E.g. corners of mouth carry much more information than haircut
Need a way to obtain basis images automatically, in order of importance!
But what’s important?

34. Principal Component Analysis
Given a point set , in an M-dim space, PCA finds a basis such that
coefficients of the point set in that basis are uncorrelated
first r < M basis vectors provide an approximate basis that minimizes the mean-squared-error (MSE) in the approximation (over all bases with dimension r) x1 x0
1st principal component
2nd principal component

35. PCA via Singular Value Decomposition
[u,s,v] = svd(A);

36. EigenFaces
First popular use of PCA on images was for modeling and recognition of faces [Kirby and Sirovich, 1990, Turk and Pentland, 1991]
Collect a face ensemble
Normalize for contrast, scale, & orientation.
Remove backgrounds
Apply PCA & choose the first N eigen-images that account for most of the variance of the data.
mean face
lighting variation

37. First 3 Shape Basis Mean appearance

38. Principal Component Analysis
Choosing subspace dimension r:
look at decay of the eigenvalues as a function of r
Larger r means lower expected error in the subspace data approximation r M 1
eigenvalues

39. Using 3D Geometry: Blinz & Vetter, 1999