Amir Hosein Omidvarnia Spring 2007 Principles of 3D Face Recognition

2 Outline Fundamentals 3D face processing stages Illumination Cone method Structured-light pattern method Elastic bunch graph method Open issues and challenges Conclusion

3 3D FRT vs. 2D FRT 2D face recognition still requires help Pose, expression, illumination variations Promises of 3D facial recognition High-security applications 3D shape information invariance Pose and illumination problems can be solved Better facial feature localization

4 Challenges in FRT The recent FERET test has revealed that there are at least two major challenges: The illumination variation problem The pose variation problem

5 Illumination variation Images of the same face appear differently due to the change in lighting Naive Solution: discarding the first few eigenfaces

6 Pose Variation Basically, the existing solution can be divided into three types: multiple images in both training stage and recognition stage multiple images in training stage, but only one image in recognition stage single image based methods

7 A Typical 3D FR System A4Vision Core Technology

8 3D Facial Recognition Pipeline Features Point Clouds Depth Images Pattern Classifier 3D Face Detection Pre-proc. Face Normalization/Alignment Fine Alignment Noise Removal Hole Filling Smoothing Cropping Landmark Finding Coarse Alignment

9 3D Face Detection This problem has not been touched so far! Simple heuristics such as nose tip In complex scenes, curvature analysis is generally used

10 Pre-processing Artifact removal Noise removal: spikes (filters), clutter (manually), noise (median filter) Holes filling (Gaussian smoothing, linear interpolation, symmetrical interpolation)

11 Face Normalization/Alignment Coarse alignment by Centre of mass, Plane fitted to the data Facial landmarks (eyes, nose tip) Fine alignment ICP (Iterative Conditional Proc.) Warping Elastic deformations

12 3D Acquisition Systems Face specific Biometrics A4Vision Geometrix Modeling Cyberware Genex Inspeck Medeim Breuckmann

13 3D Face Databases UND 275 subjects, 943 scans Shape + texture FRGC 400 subjects, 4007 scans Shape + texture 3D_RMA 120 subject, 6 scans Shape only GavabDB 61 subjects (9 scans) Shape only Pose, expression variations USF database 357 scans 3DFS generator Custom face databases 12 persons to ~6000 persons (A4Vision) UND GavabDB 3DFS

14 3D Face Recognition Approaches Appearance-Based Methods Feature-Based Methods Model-Based Methods

15 An Appearance-Based Method Illumination Cone Method

16 Lambertian Model Lambertian shading assumes that the incoming light is reflected equally in all directions, without bias. The angle of incoming light has no effect on the direction in which it is reflected. Lambertian Phong

17 Illumination cone For a Lambertian surface: Image x superpositioned with k light sources can be written as:

18 Illumination cone Database images of Yale University: Different Illuminations Different Poses

19 Illumination cone Image Acquisition

20 Illumination cone Least Square estimation is used to find normal unit vectors.

21 Illumination cone Illumination cone is a subspace covers the variation in illumination. Basis images Synthetic images Reconstructed surface by means of GBR Ambiguity

22 Illumination cone 1.Representations and Algorithms for Face Recognition 2.Constructing 117 (19x7) different poses by means of planar transformations (non-linear warping) 3.Constructing the Illumination Cone of each pose from different lighting conditions

23 Illumination cone 4.Decreasing the number of lighting conditions using PCA dimension reduction down to 11 5.Applying SVD based methods to reduce the number of database images (11x117) down to These 100 images form the image basis space for each person

24 A Feature-Based Method Structured-Light Pattern Method

25 Structured-Light Surface Rendering

26 Structured-Light Surface Rendering Striped images Reconstructed surface

27 Structured-Light Surface Rendering Curvature Analysis for Surface Matching

28 Triangulation-Interpolation

29 A Model-Based Method Elastic Bunch Graph Matching

30 Elastic Bunch Graph use Gabor wavelet transform to extract face features so that the recognition performance can be invariant to the variation in poses.

31 Elastic Bunch Graph Gabor wavelet decomposition Gabor kernels

32 Gabor Filters

33 Jets Small patch gray values Wavelet transform

34 Comparing Jets Amplitude similarity Phase similarity

35 Comparing Jets

36 Face Bunch Graphs (FBG) Stack like general representation Graph similarity function

37 Graph Extraction Step 1: find approximate face position Step 2: refine position and size Step 3: refine size and find aspect ratio Step 4: local distortion

38 Recognition Comparing image graph Recognized for highest similarity

39 Open Issues & Challenges Uncontrolled acquisition Non-cooperative Different lighting conditions Texture map + shape map inconsistencies Real-time 3D video data Computational complexity Issues related to performance assessment Publicly available standard face databases Quality (resolution) of the data Artifacts such as eyeglasses

40 Conclusions 3D face recognition systems were proposed to overcome expression, illumination, and pose challenges Illumination correction is simpler Facial landmark localization is better The core algorithm, ICP, has limited capabilities Not suitable for non-rigid deformations