AAM based Face Tracking with Temporal Matching and Face Segmentation Dalong Du

Outline Author Introduction AAM Introduction Abstract Method and Theory Experiment

Author Introduction Mingcai Zhou – Institute of Automation Chinese Academy of Sciences Lin Liang – Microsoft Research Asia –

Author Introduction Jian Sun – Microsoft Research Asia joined in July, – Educational background BS degree, MS degree and Ph.D degree from Xian Jiaotong University in 1997, 2000 and 2003 – Current research interests Interactive compute vision (user interface + vision) Internet compute vision (large image collection + vision) stereo matching and computational photography

Author Introduction Yangsheng Wang – Director of Digital Interactive Media Lab, Institute of Automation Chinese Academy of Sciences – Educational background BS degree, MS degree and Ph.D degree from Huazhong University of Science and Technology

AAM Introduction Shape Model Appearance (Texture) Model AAM Model Search

AAM—Shape Model Face Q consists of N landmark points – – The geometry information of Q decouples into two parts: A shape S – Shape is the geometric information invariant to a particular class of transformations – e.g. Or other linear or nonlinear methods A transformation – θ – e.g. similarity s, R, t Or Affine or others. – Similarity » b θ x = (x 1,y 1, …, x n, y n ) T Same shape Different shape

AAM—Shape Model Shape Model Building – Given a set of shapes – Align shapes into common frame Procrustes analysis – Estimate shape distribution p(x) Use PCA The aligned shapes

AAM—Shape Model Shape Model Building, continued – Given aligned shapes, { } – Apply PCA Compute mean and eigenvectors of covar. – P – First t eigenvectors of covar. matrix – b – Shape model parameters

AAM—Texture Model Building Texture Models – For each example, extract texture vector – Normalise vectors (as for eigenfaces) – Build eigen-model Texture, g Warp to mean shape

AAM—Texture Model Warp method

AAM—Texture Model Warp method, continued

AAM—Model Search Find the optimal shape parameters and appearance parameters to minimize the difference between the warped-back appearance and synthesized appearance map every pixel x in the model coordinate to its corresponding image point Computed by the inverse Compositional parameter Update technique

Abstract Problems – Generalization problem – images with cluttered background How to do? – A temporal matching constraint in AAM fitting Enforce an inter-frame local appearance constraint between frames – Introduce color-based face segmentation as a soft constraint

Method and Theory Extend basic AAM to Multi-band AAM – The texture(appearance) is a concatenation of three texture band values The intensity (b) X-direction gradient strength (c) Y-direction gradient strength (d)

Method and Theory Temporal Matching Constraint – Select feature points with salient local appearances at previous frame – Optimize the shape parameters to match the local appearances at current frame

Method and Theory Temporal Matching Constraint, continued – : a set of feature points Selected by a corner detector and some semantic points – : the face appearance of frame t-1 – : the local patch corresponding to the j-th feature point – : the average intensity of j-th patches of frame t-1 and t respectively Normalize the illuminations of two patches

Method and Theory Temporal Matching Constraint, continued – Add a new term to the AAM cost function Empirically, Can be efficiently minimized based on inverse compositional algorithm

Method and Theory Temporal Matching Constraint, continued – Be resistant to global illumination changes Match local patches – Do not suffer from the mismatched points Feature matching is continuously refined by updating the shape parameters during AAM fitting

Method and Theory Initialize shape – Good initial parameters -> good AAM fitting – Method Selected feature points at frame t-1 Matched feature points at frame t Remaining feature points after main direction filter

Method and Theory Initialize shape, continued – M matched points – Estimate the initial shape parameters represents the consistency of feature points I’s direction is the estimated position of the point I given the shape parameters p are the vertex coordinate of the triangle are the triangle coordinate Gauss-Newton algorithm

Method and Theory Face Segmentation Constrained AAM – Problem: AAM tends to fit the face outline to the background edges – Method: segment the face region using an adaptive color model and constrain AAM fitting

Method and Theory Formalization – Where are the locations of the selected outline points in the model coordinate Wc = 0.01

Experiments RI: robust initialization TO: temporal matching constraint FS: face segmentation

Experiments

Thank you