2011/5/23 3D Morphable Model Based Face Replacement in Video Presenter ： Bo-Hung Chen Adviser ： Dr. Shih-Chung Chen Yi-Ting Cheng, Virginia Tzeng, Yung-Yu Chuang, Ming Ouhyoung Dept. of Computer Science and Information Engineering National Taiwan University (2008)

2011/5/23 Outline Motivation Introduction System structure 3D Face Database Expression Model Database Estimation, Correction and Compared Results References

2011/5/23 Motivation Entertainment To assess Plastic surgery Why this paper?

2011/5/23 In the past, The most naive method is to ask the source subject to act the same as the target subject under the similar lighting condition In this paper, we present a system for face replacement in video to replace the target subject face in the target video with the source subject face, under similar pose Introduction

2011/5/23 System structure

2011/5/23 3D Face Database The morphable model of 3D faces is a vector space of 3D shapes and textures spanned by a set of example faces This morphable model is derived from structural light 3D scans of 117 adults (92 males and 25 females). A correspondence algorithm makes all the faces fully correspondent. Each 3D face model is represented by vertices with textures.

2011/5/23 Profile line of the model Furthermore, to match the profile line of the model better, we define an energy term to match the profile line between the height of eyes and jaw. At each horizontal scan line, an energy term restricts the x coordinate of the profile line in I input close to the one in I model : Where x p,i and x’ p,i are the x coordinate of the profile line on the i-th horizontal scan line in I input and I model respectively Energy term ：能 [ 量 ] 項 ( 物理學專有名詞 ) Norm : 範數，是具有「長度」概念的函數，為向量空間內的所有向量賦予非零的正長度或大小。

2011/5/23 Principal Component Analysis PCA is used to perform a basis transformation to an orthogonal coordinate system formed by the eigenvectors e S,i and e T,i in descending order according to the eigenvalues σ S,i and σ T,i of the covariance matrices

2011/5/23 Expression Database We have to extend the neutral source face model to expressional model. 13 key expressions : –Emotional expressions : angry, smiling, happy, sad, and surprised. –Verbal key expressions : pronouncing ‘a’, ‘e’, ‘uh’, ‘m’, ‘o’, and ‘u’. –Expressions of closing eyes (Left and Right) Each model in the expression database is represented by 436 vertices, and we manually match these vertices to the vertices of the reconstructed model.

2011/5/23 Estimation, Correction and Compared In this section, we introduce the head pose estimator first, and then we introduce the lighting estimator and face relighting module in detail.

2011/5/23 Estimation Euclidean Distance Given an input image, the goal is to minimize the Euclidean distance over all color channels and all pixels between the input image I input and the image I model synthesized from the current model. To match the geometry of model better, we exploit the labeled feature points (q x,i, q y,i ) and the image-plane position (p x,ki, p y,ki ) of the corresponding vertices k i in an additional feature term.

2011/5/23 Pose Estimation Based on the set of 87 feature points which are detected by the face alignment module and the corresponding preselected feature points in the target face model, we can estimate pose parameters by minimizing the error E between them. Where w i is the weight of the i-th feature point, (q x,i, q y,i ) is the position of the i-th feature point of face alignment, and (p x,i, p y,i ) is the project position of the i-th feature point of the target model.

2011/5/23 Corrected Overfitting E ff ects Minimization of these energy functions with respect to α, β, ρ may cause overfitting e ff ects. Therefore, we employ a maximum a posteriori estimator (MAP). Finally, the posteriori probability is then maximized by minimizing Posteriori probability ： 後端驗證概率 ( 物理學專有名詞 ) Maximum a posteriori estimator (MAP) ：極端後端驗證

2011/5/23 Face Relighting If the source face and the target face were under di ff erent illumination, the replacement result would appear perceptually unreasonable, so we need to adjust skin color and lighting of the source face where ρ c is the average color of each color channel, a c,k are the spherical harmonic coe ffi cients which we want to estimate as lighting parameters, H are the spherical harmonics, and n(x, y) is the surface normal at the image location (x, y). We use the 3D face models to render normal maps of both the target face and source face

2011/5/23 Matching the Mouth and Blinking

2011/5/23 Result

2011/5/23 Results

2011/5/23 References [1] The prestige, [2] R. Basri and D. W. Jacobs. Lambertian reflectance and linear subspaces. IEEE Transactions on Pat- tern Analysis and Machine Intelligence, 25(2):218– 233, [3] D. Bitouk, N. Kumar, S. Dhillon, P. N. Belhumeur, and S. K. Nayar. Face swapping: automatically replacing faces in photographs. ACM Transactions on Graphics (SIGGRAPH), 27(3), [4] V. Blanz, C. Basso, T. Poggio, and T. Vetter. Re- animating faces in images and video. Computer Graphics Forum, 22:641–650, [5] V. Blanz, K. Scherbaum, T. Vetter, and H.-P. Sei- del. Exchanging faces in images. Computer Graph- ics Forum, 23(3):669–676, [6] V. Blanz and T. Vetter. A morphable model for the synthesis of 3d faces. In Computer Graphics Proc. SIGGRAPH’99, pages 187–194, 1999.

2011/5/23 [7] Y. Liang. Image based face replacement in video. Master’s thesis, CSIE Department, National Tai- wan University, [8] Z. Liu, Z. Zhang, C. Jacobs, and M. Cohen. Rapid modeling of animated faces from video images. In Proceedings of ACM International Conference on Multimedia, pages 475–476, [9] J. A. Nelder and R. Mead. A simplex method for function minimization. Computer Journal, 7:308– 313, [10] P. P´erez, M. Gangnet, and A. Blake. Poisson im- age editing. ACM Transactions on Graphics (SIG- GRAPH), 22:313–318, [11] F. Pighin, J. Hecker, D. Lischinski, R. Szeliski, and D. H. Salesin. Synthesizing realistic facial expres- sions from photographs. In SIGGRAPH ’06: ACM SIGGRAPH 2006 Courses, page 19, [12] H. Pyun, Y. Kim, W. Chae, H. W. Kang, and S. Y. Shin. An example-based approach for facial expression cloning. In 2003 ACM SIGGRAPH / Eurographics Symposium on Computer Animation, pages 167–176, [13] J. yong Noh and U. Neumann. Expression cloning. In SIGGRAPH ’06: ACM SIGGRAPH 2006 Courses, page 22, 2006.

2011/5/23 Thanks for your attention