Unsupervised Face Alignment by Robust Nonrigid Mapping Experimental Results 1. Experimental Testbed Google Image Dataset 3467 face images collected by Google image search 101 individuals, 17-62 images for each individual LFW dataset Oxford VGG facial feature locator PC with Intel Core-2 Duo 2G Hz CPU and 2GB RAM Http://www.vision.ee.ethz.ch/~zhuji/facealign/index.html 2. Face Alignment Unsupervised Face Alignment by Robust Nonrigid Mapping Jianke Zhu¹, Luc Van Gool¹², Steven C.H. Hoi³ ¹ETH Zurich, ²K. U. Leuven, ³Nanyang Technological University Overview and Motivation Related Work Supervised Face Alignment Active appearance models, T. Cootes et al. TPAMI’01. Generalized shape regularization model, L. Gu and T. Kanade ECCV’08 Congealing Learned-Miller’s congealing, G.B. Huang et al. ICCV’07. Least square congealing, M. Cox et al. CVPR’08. Patch-based Rectification Kanade-Yamada algorithm, CIRA’03. Viewpoint invariant face recognition, A. Ashraf et al. CVPR’08. Nonrigid Shape Recovery Nonrigid surface detection, Pilet et al. CVPR’05, J. Zhu et al.CVPR’07 Deformable Lucas-Kanade algorithm, J. Zhu et al. TPAMI’09. Fig 3. Average of aligned facial images 3. Numerical Evaluation on Face Recognition Fig.1 We first fit a set of template images to the input image by a robust deformable Lucas-Kanade fitting scheme. Then, the input face image is rectified into the canonical frontal view. Fig. 4 Comparison of face alignment for various head poses. We show the results of the congealing, CMU’s online system , and the proposed joint deformable Lucas-Kanade method. Contributions of Unsupervised Face Alignment A new promising approach for face alignment Avoid the needs of manual labeling Only make assumption on the parameterization Flexible and practical for real applications Fig. 6 Correct recognition rate on 3467 Google face images (Intensity) Unsupervised Face Alignment Problem Formulation Fig.2 Example of image-to-image face alignment. We overlay a triangular mesh (Fig. 2(a)) and directly parameterize the transformation by the mesh vertex coordinates s. With its 2N free variables, the problem is ill-posed. To overcome this challenge, we adopt the algorithm proposed by Zhu et al. TPAMI’09 by introducing a regularization term: where is regularization coefficient, is the increments of mesh vertices and K is a sparse banded stiffness matrix. The solution for above optimization can be computed as below: where H is the Hessian matrix, and update equation: Robust Deformable Lucas-Kanade Algorithm To effectively handle outliers in intensities, we apply a robust estimator to the image differences u: Joint Face Alignment To increase the chances of homing in on the solution, we introduce multiple template images into the energy function. These jointly drive forward the evolution of the mesh on the input – i.e. target – face. We have the following energy function for joint face alignment: To preserve the texture variations, we employ the steepest descent matrices computed from each of the template images: where Ai is the steepest descent image, and Hessian matrix is computed as follows: Fig. 7 ROC curves on LFW dataset. 4. Face Swapping Application Fig. 5 Comparison of face alignment results for various expressions. ~ Conclusions A novel unsupervised face alignment method without assuming a rigid affine transformation for alignment A robust and efficient optimization scheme to make the solution reliable against outliers; A joint face alignment scheme that allows to use multiple templates to deal with large appearance changes Experiments on large sets of images, comparing against state-of-the-art face alignment techniques The Twelfth IEEE International Conference on Computer Vision (ICCV2009)