Real-time Combined 2D+3D Active Appearance Models Jing Xiao, Simon Baker,Iain Matthew, and Takeo Kanade CVPR 2004 Presented by Pat Chan 23/11/2004

Outline Introduction Active Appearance Models AAMs 3D Morphable Models 3DMMs Representational Power of AAM Combined 2D+3D AAMs Conclusion

Introduction Active Appearance Models are generative models commonly used to model faces Another closely related type of face models are 3D Morphable Models In this paper, it tries to model 3D phenomena by using the 2D AAM Constrain the AAM with the 3D models to achieve a real-time algorithm for fitting the AAM

Active Appearance Models (AAMs) 2D linear shape is defined by 2D triangulated mesh and in particular the vertex locations of the mesh. Shape s can be expressed as a base shape s 0. p i are the shape parameter. s 0 is the mean shape and the matrices s i are the eigenvectors corresponding to the m largest eigenvalues

Active Appearance Models (AAMs) The appearance of an independent AAM is defined within the base mesh s 0. A(u) defined over the pixels u ∈ s 0 A(u) can be expressed as a base appearance A 0 (u) plus a linear combination of l appearance Coefficients λ i are the appearance parameters. A 0 (u) A 1 (u) A 2 (u) A 3 (u)

Active Appearance Models (AAMs) The AAM model instance with shape parameters p and appearance parameters λ is then created by warping the appearance A from the base mesh s 0 to the model shape s. Piecewise affine warp W(u; p): (1) for any pixel u in s 0 find out which triangle it lies in, (2) warp u with the affine warp for that triangle. M(W(u;p))

Fitting AAMs Minimize the error between I (u) and M(W(u; p)) = A(u). If u is a pixel in s 0, then the corresponding pixel in the input image I is W(u; p). At pixel u the AAM has the appearance At pixel W(u; p), the input image has the intensity I (W(u; p)). Minimize the sum of squares of the difference between these two quantities: u uuu

DEMO Video – 2D AAMs

3D Morphable Models (3DMMS) 3D shape of 3DMM is defined by 3D triangulated mesh and in particular the vertex location of the mesh. The ŝ can be expressed as a based shape ŝ 0 plus a linear combination of m shape matrices ŝ i :

3D Morphable Models (3DMMS) The appearance of a 3DMM is defined within a 2D triangulated mesh that has the same topology as the base mesh ŝ 0. The appearance Â(u) can be expressed as a based appearance  0 (u) plus a linear combination of l appearance images  i (u).

3D Morphable Models (3DMMS) To generate a 3DMM model instance, an image formation model is used to convert the 3D shape ŝ in to 2D mesh. The result of the imaging 3D point x = (x, y, z) T is: i, j are the projection axes, o is the offset of the origin Given shape parameters p i  compute 3D shape  map to 2D  compute appearance  warp onto 2D mesh (defined by mapping from 2D vertices in ŝ 0 to 2D vertices for 3D ŝ.)

Representational Power of AAM Can 2D shape models represent 3D? The 2D shape variation of the 3D model: The projection matrix can be expressed as: Therefore 3D model can be represented as combination of: The variation of the 3D model can therefore be represented by an appropriate set of 2D shape vectors, such as: 6*(m^+1) 2D shape vectors needed to represent a 3D model

Representational Power of AAM Experiments  Use 3D-cube : ŝ = ŝ 0 + p 1 ŝ 1  Generate 60 sets p 1 and P randomly  Synthesize 2D shapes of 60 3D model instances  Compute 2D shape model by performing PCA on 60 2D shapes  Result: 12 shapes vectors for each 2D shape mode  Confirm: 6*(m^+1) 2D vector is required However, 2D models generate invalid cases. Constrains is need to add on the model

Combined 2D + 3D AAMs At time t, we have 2D AAM shape vector in all N images into a matrix: Represent as a 3D linear shape modes W = MB =

Compute the 3D Model Perform singular value decomposition (SVD) on W and factorize it into: The scaled projection matrix M and the shape vector matrix B are given by: G is the corrective matrix. Additional rotational and basis constrain to compute G  M and B can be determined Thus, the 3D shape modes can be computed from the 2D AMM shape modes and the 2D AAM tracking results.

Calculate the Corrective Matrix Rotational constraints and basis constraints are used. Rotational constraints (denote GG T by Q): where ˜M 2*i−1:2*i represents the i th two-row of ˜M c is the coefficient and R is rotation matrices Due to orthogonormality of rotation matrices and Q is symmetric,

Calculate the Corrective Matrix Basis constraints: We find K frames including independent shapes and treat those shapes as a set of bases, the bases are determined uniquely, we have

Compute the 3D Model AAM shapes AAM appearance First three 3D shapes modes

Constraining an AAM with 3D Shape Constraints on the 2D AAM shape parameters p = (p 1, …, p m ) that force the AAM to only move in a way that is consistent with the 3D shape modes: and the 2D shape variation of the 3D shape modes over all imaging condition is: Legitimate values of P and p such that the 2D projected 3D shape equals the 2D shape of AAM. The constraint is written as:

Fitting with 3D Shape Constraints AAM fitting is to minimize: I.e the error between the appearance and the original image Impose the constrains of 2D projected 3D shape equals the 2D shape of AAM as soft constrains on the above equation with a large K:

Fitting with 3D Shape Constraints Optimize for the AAM shape p, q, and the appearance λ parameters: Calculate the square difference between the appearance and the original image and project the difference into orthogonal complement of the linear subspace spanned by the vectors A 1, …, A l. It is optimized by using inverse compositional algorithm, I.e. iteratively minimizing: Then, solve the appearance parameters using the linear closed form solution:

Experimental Results Initialization After 30 IterationsConverged Estimates of the 3D Pose extracted from the current estimate of the camera matrix P 2D AAM Estimated 3D shape

Experimental Results Results of using the algorithm to track a face in 180 frame video sequence by fitting the model in each frame

DEMO Video -- 2D+3D AAMs

2D+3D AAM Model Reconstruction Input Image Tracked result (2D+3D fit result) 2D+3D model reconstruction Shows two new view reconstruction

Compare the fitting speed with 2D AAMs Frames per second of 2D+3D > 2D AAM Iteration per second of 2D > 2D+3D, but 2D need more iteration for convergence

Conclusion 2D AAMs can represent any phenomena that 3DMMs can. Showed how to compute the equivalent 3D shape models from a 2D AAM with basis constrains, rotational constrains. Improve the fitting speed of the 2D AAMs with 3D shapes constrains 2D + 3D AAM is the ability to render the 3D model from novel viewpoint.

Q & A