1 Bronstein 2 & Kimmel An isometric model for facial animation and beyond AMDO, Puerto de Andratx, 2006 An isometric model for facial animation and beyond Michael M. Bronstein Department of Computer Science Technion – Israel Institute of Technology

2 Bronstein 2 & Kimmel An isometric model for facial animation and beyond AMDO, Puerto de Andratx, 2006 Co-authors Ron KimmelAlex Bronstein

3 Bronstein 2 & Kimmel An isometric model for facial animation and beyond AMDO, Puerto de Andratx, 2006 Agenda Single texture mapping onto an animated face Morphing Expression interpolation and extrapolationBeyond…

4 Bronstein 2 & Kimmel An isometric model for facial animation and beyond AMDO, Puerto de Andratx, 2006 Isometric model of facial expressions Face: deformable Riemannian surface with geodesic distances Facial expression: approximate isometry B 2 K, IJCV 2005

5 Bronstein 2 & Kimmel An isometric model for facial animation and beyond AMDO, Puerto de Andratx, 2006 Virtual makeup Map a single texture image onto a 3D video sequence of animated face in an expression-invariant manner TEXTURE3 D V I D E O S E Q U E N C E

6 Bronstein 2 & Kimmel An isometric model for facial animation and beyond AMDO, Puerto de Andratx, 2006 Approach I: Common parametrization Parametrize and over a common parametrization domain by the maps and Draw the texture in the parametrization domain Map the texture to and using the maps and

7 Bronstein 2 & Kimmel An isometric model for facial animation and beyond AMDO, Puerto de Andratx, 2006 How to find a parametrization? G. Zigelman et al., IEEE TVCG, 2002 Embed and into the plane by a minimum-distortion map

8 Bronstein 2 & Kimmel An isometric model for facial animation and beyond AMDO, Puerto de Andratx, 2006A. Elad, R. Kimmel, CVPR 2001 Given a sampling the minimum-distortion embedding is found by optimizing over the images and not on itself Multidimensional scaling Approximately common parametrization Requires alignment (usually manual, according to some fiducial points) Difficult to handle different or complicated topologies Alternative, more robust formulation:

9 Bronstein 2 & Kimmel An isometric model for facial animation and beyond AMDO, Puerto de Andratx, 2006 Approach II: Correspondence problem Assume that the texture is drawn on Find correspondence between and Transfer the texture by the map In case of common parametrization,

10 Bronstein 2 & Kimmel An isometric model for facial animation and beyond AMDO, Puerto de Andratx, 2006 How to find the correspondence? Embed into by a minimum-distortion map Fiducial points-based methods usually give sparse correspondence and require manual assistance Optical flow between texture images (Blanz et al.) is not applicable when only geometric information is given

11 Bronstein 2 & Kimmel An isometric model for facial animation and beyond AMDO, Puerto de Andratx, 2006 Generalized multidimensional scaling (I) B 2 K, PNAS 2006 G MDS: are computed once using fast marching have to be computed at each iteration Note that are not restricted to the mesh vertices

12 Bronstein 2 & Kimmel An isometric model for facial animation and beyond AMDO, Puerto de Andratx, 2006 Generalized multidimensional scaling (II) More robust in practice Weights allow to handle different topologies (e.g. open mouth) and missing data (scanner artifacts) Multiresolution / multigrid schemes to prevent local convergence B 2 K, PNAS 2006 A weighted least-squares version of the problem

13 Bronstein 2 & Kimmel An isometric model for facial animation and beyond AMDO, Puerto de Andratx, 2006 ReferenceTransferred texture

14 Bronstein 2 & Kimmel An isometric model for facial animation and beyond AMDO, Puerto de Andratx, 2006 Calculus of faces (I) Interpolation Extrapolation Abstract manifold of facial articulations Face animation: trajectory Minimum-distortion correspondence allows creating a (locally) linear space, in which faces are represented as vectors

15 Bronstein 2 & Kimmel An isometric model for facial animation and beyond AMDO, Puerto de Andratx, 2006 Calculus of faces (II) Extrinsic coordinates and texture interpolation CORRESPONDENCE Extrinsic geometry Texture

16 Bronstein 2 & Kimmel An isometric model for facial animation and beyond AMDO, Puerto de Andratx, 2006 Interpolation I N T E R P O L A T E D F R A M E S Temporal super-resolution: increase frame rate of 3D video by adding interpolated frames Interpolation of geometry and texture

17 Bronstein 2 & Kimmel An isometric model for facial animation and beyond AMDO, Puerto de Andratx, 2006 Extrapolation Expression exaggeration: synthesize new expressions using a non-convex combination Interpolation of geometry and texture NEUTRALEXPRESSIONEXAGGERATED EXPRESSION

18 Bronstein 2 & Kimmel An isometric model for facial animation and beyond AMDO, Puerto de Andratx, 2006 Bronstein 2 & Kimmel An isometric model for facial animation and beyond

19 Bronstein 2 & Kimmel An isometric model for facial animation and beyond AMDO, Puerto de Andratx, 2006 Morphing Convex combination between two different faces Morphing of geometry and texture SOURCETARGET

20 Bronstein 2 & Kimmel An isometric model for facial animation and beyond AMDO, Puerto de Andratx, 2006 Bronstein 2 & Kimmel An isometric model for facial animation and beyond

21 Bronstein 2 & Kimmel An isometric model for facial animation and beyond AMDO, Puerto de Andratx, 2006 Virtual body art Texture mapping on articulated human body, similarly to body art

22 Bronstein 2 & Kimmel An isometric model for facial animation and beyond AMDO, Puerto de Andratx, 2006 ReferenceTransferred texture 22 Bronstein 2 & Kimmel An isometric model for facial animation and beyond

23 Bronstein 2 & Kimmel An isometric model for facial animation and beyond AMDO, Puerto de Andratx, ReferenceTransferred texture Bronstein 2 & Kimmel An isometric model for facial animation and beyond

24 Bronstein 2 & Kimmel An isometric model for facial animation and beyond AMDO, Puerto de Andratx, 2006 Summary Isometric model of facial expressions Automatic dense correspondence based on the minimum-distortion mapping Possibility to find correspondence between partially missing or partially overlapping surfaces (COME TO THE SECOND TALK AT 15:30) Texture mapping, expression synthesis, morphing, etc… GMDS - a generic tool that can be applied to different problems