1. Unsupervised Face Alignment by Robust Nonrigid Mapping
Experimental Results 1. Experimental Testbed
Google Image Dataset
3467 face images collected by Google image search
101 individuals, images for each individual
LFW dataset
Oxford VGG facial feature locator
PC with Intel Core-2 Duo 2G Hz CPU and 2GB RAM
2. Face Alignment
Unsupervised Face Alignment by Robust Nonrigid Mapping
Jianke Zhu¹, Luc Van Gool¹², Steven C.H. Hoi³
¹ETH Zurich, ²K. U. Leuven, ³Nanyang Technological University
Overview and Motivation
Related Work
Supervised Face Alignment
Active appearance models, T. Cootes et al. TPAMI’01.
Generalized shape regularization model, L. Gu and T. Kanade ECCV’08
Congealing
Learned-Miller’s congealing, G.B. Huang et al. ICCV’07.
Least square congealing, M. Cox et al. CVPR’08.
Patch-based Rectification
Kanade-Yamada algorithm, CIRA’03.
Viewpoint invariant face recognition, A. Ashraf et al. CVPR’08.
Nonrigid Shape Recovery
Nonrigid surface detection, Pilet et al. CVPR’05, J. Zhu et al.CVPR’07
Deformable Lucas-Kanade algorithm, J. Zhu et al. TPAMI’09.
Fig 3. Average of aligned facial images
3. Numerical Evaluation on Face Recognition
Fig.1 We first fit a set of template images to the input image by a robust deformable Lucas-Kanade fitting scheme. Then, the input face image is rectified into the canonical frontal view.
Fig. 4 Comparison of face alignment for various head poses. We show the results of the congealing, CMU’s online system , and the proposed joint deformable Lucas-Kanade method.
Contributions of Unsupervised Face Alignment
A new promising approach for face alignment
Avoid the needs of manual labeling
Only make assumption on the parameterization
Flexible and practical for real applications
Fig. 6 Correct recognition rate on 3467 Google face images (Intensity)
Unsupervised Face Alignment
Problem Formulation
Fig.2 Example of image-to-image face alignment.
We overlay a triangular mesh (Fig. 2(a)) and directly parameterize the transformation by the mesh vertex coordinates s. With its 2N free variables, the problem is ill-posed. To overcome this challenge, we adopt the algorithm proposed by Zhu et al. TPAMI’09 by introducing a regularization term:
where is regularization coefficient, is the increments of mesh vertices and K is a sparse banded stiffness matrix.
The solution for above optimization can be computed as below:
where H is the Hessian matrix, and update equation:
Robust Deformable Lucas-Kanade Algorithm
To effectively handle outliers in intensities, we apply a robust estimator to the image differences u:
Joint Face Alignment
To increase the chances of homing in on the solution, we introduce multiple template images into the energy function. These jointly drive forward the evolution of the mesh on the input – i.e. target – face. We have the following energy function for joint face alignment:
To preserve the texture variations, we employ the steepest descent matrices computed from each of the template images:
where Ai is the steepest descent image, and Hessian matrix is computed as follows:
Fig. 7 ROC curves on LFW dataset.
4. Face Swapping Application
Fig. 5 Comparison of face alignment results for various expressions. ~
Conclusions
A novel unsupervised face alignment method without assuming a rigid affine transformation for alignment
A robust and efficient optimization scheme to make the solution reliable against outliers;
A joint face alignment scheme that allows to use multiple templates to deal with large appearance changes
Experiments on large sets of images, comparing against state-of-the-art face alignment techniques
The Twelfth IEEE International Conference on Computer Vision (ICCV2009)