1. Last tuesday, you talked about active shape models Data set of 1,500 hand-labeled faces 20 facial features (eyes, eye brows, nose, mouth, chin) Train 40 individual regressors (x and y positions for each facial feature) 1

2. Problems/Solutions Our regressors aren’t perfect, but they’re often wrong in conflicting ways Facial feature positions aren’t independent of each other – eg. knowing something about the position of the eyes ought to provide a clue about the position of the nose Face alignment needs to be more than the sum of its parts – Active shape model

3. Shape model Given many examples of face shapes (and ignoring the image data) Find a lower dimensional model that can still represent the high dimensional examples 3

4. The Active Shape Model framework Build a model of face shapes to go along with our facial feature models, then combine the two into one output ShapeFeatures Alignment 4

5. Shape model Problem: We tried to describe our face using a 40-dimensional vector. That gives us a little bit too much flexibility: Solution: If 40 dimensions are too many, let’s build a low-dimensional model FaceDog (?)

6. Learning a shape model Use Principal Component Analysis (PCA) on the feature positions – Represent each example as a 40-dimensional vector, (x 1, y 1, x 2, y 2, …, x 20, y 20 ) – Subtract the mean from each vector – Find the most important (principal) axes PCA gives us a total of 40 axes, but if we only select a few of them (say, the top six), we’ll get a low dimensional parameterization

7. Learning a shape model Compute mean face shape

8. Learning a shape model Compute principal components 8

9. ACTIVE APPEARANCE MODELS “Interpretation through synthesis” Form a model of the object/image (Learnt from the training dataset) I. Matthews and S. Baker, "Active Appearance Models Revisited," International Journal of Computer Vision, Vol. 60, No. 2, November, 2004, pp. 135 - 164.

10. Face appearance has two components: Model “Shape” “texture”

11. Active Shape Models training set

12. (Hard to automate) Shape Vector provides alignment = 42 Alexei (Alyosha) Efros, 15-463 (15-862): Computational Photography, http://graphics.cs.cmu.edu/courses/15-463/2005_fall/www/Lectures/faces.ppt

13. Random Aside (can’t escape structure!) Alexei (Alyosha) Efros, 15-463 (15-862): Computational Photography, http://graphics.cs.cmu.edu/courses/15-463/2005_fall/www/Lectures/faces.ppt Antonio Torralba & Aude Oliva (2002) Averages: Hundreds of images containing a person are averaged to reveal regularities in the intensity patterns across all the images.

14. Random Aside (can’t escape structure!) Tomasz Malisiewicz, http://www.cs.cmu.edu/~tmalisie/pascal/trainval_mean_large.png

15. “100 Special Moments” by Jason Salavon Jason Salavon, http://salavon.com/PlayboyDecades/PlayboyDecades.shtml

16. Given Shape Vectors (positions of set of keypoints) Shape Vector provides alignment = 42 Alexei (Alyosha) Efros, 15-463 (15-862): Computational Photography, http://graphics.cs.cmu.edu/courses/15-463/2005_fall/www/Lectures/faces.ppt Collect keypoints from training data in M s whose columns are training shapes. Subtract mean shape (s), then run SVD to get (M s = U s S V s ’). s i = s + U s a i

17. “Shape free” Texture Models warp to mean shape

18. Once you have the shape model for each image, warp the model to the mean configuration: Now, all faces are the same size and shape and place. So you can create a column vector out of the relevant pixels. Put all those columns together (one for each training image) into M t Subtract mean texture (t), then run SVD to get M t = U t S V t ’. t i = t + U t b i

19. Face model All faces parameterized by mean texture t and mean shape s. Each face has shape parameter a i, and texture paramter b i. Given a shape and a texture, we construct a face as: – Texture = t + U t b – Shape = s + U s a – Make the texture, then warp it onto the new shape. – Awkward Mathematical notion. – Warp(Image, shape parameters) – W(t + U t b, s + U s a)

20. PCA Craziness The complete description of the face is [a i,b i ]. Let’s put those into one column, make a matrix M for all those columns and run PCA. Why the heck would we do this?

21. Average faces. Alignment is the key! 1. Warp to mean shape 2. Average pixels Alexei (Alyosha) Efros, 15-463 (15-862): Computational Photography, http://graphics.cs.cmu.edu/courses/15-463/2005_fall/www/Lectures/faces.ppt

22. Random Aside Enhancing Gender more same original androgynous more opposite D. Rowland, D. Perrett. “Manipulating Facial Appearance through Shape and Color”, IEEE Computer Graphics and Applications, Vol. 15, No. 5: September 1995, pp. 70-76

23. Playing with the Parameters First two modes of shape variationFirst two modes of gray-level variation First four modes of appearance variation

24. Essence of the Idea (cont.) Explain a new example in terms of the model parameters

25. Active Appearance Model Search (Results)

26. Active Appearance Model Search Given: Full training model set, new image to be interpreted, “reasonable” starting approximation Goal: Find model with least approximation error High Dimensional Search: Curse of the dimensions strikes again

27. Active Appearance Model Search Trick: Each optimization is a similar problem, can be learnt Assumption: Linearity Perturb model parameters with known amount Generate perturbed image and sample error Learn multivariate regression for many such perterbuations

28. Active Appearance Model Search Algorithm: current estimate of model parameters: normalized image sample at current estimate

29. Active Appearance Model Search Slightly different modeling: Error term: Taylor expansion (with linear assumption) Min (RMS sense) error: Systematically perturb and estimate by numerical differentiation

30. Final Projects 1 person: Implement any recent paper from CVPR/ ECCV/ ICCV, and apply to some new data. 2-3 people, small research project; perhaps starting with recent paper and substantial extension, or one of the following ideas

31. Single Image Haze Removal Using Dark Channel Prior – http://research.microsoft.com/en- us/um/people/jiansun/papers/deh aze_cvpr2009.pdf http://research.microsoft.com/en- us/um/people/jiansun/papers/deh aze_cvpr2009.pdf Example 1 person projects…

32. Example 2-3 person projects Matching historical and modern photos http://www.thirdview.org/3v/rephotos/index. html http://www.thirdview.org/3v/rephotos/index. html

33. What year was this photo taken?