2. Amir Hosein Omidvarnia Spring 2007 Principles of 3D Face Recognition

3. 2 Outline Fundamentals 3D face processing stages Illumination Cone method Structured-light pattern method Elastic bunch graph method Open issues and challenges Conclusion

4. 3 3D FRT vs. 2D FRT 2D face recognition still requires help Pose, expression, illumination variations Promises of 3D facial recognition High-security applications 3D shape information invariance Pose and illumination problems can be solved Better facial feature localization

5. 4 Challenges in FRT The recent FERET test has revealed that there are at least two major challenges: The illumination variation problem The pose variation problem

6. 5 Illumination variation Images of the same face appear differently due to the change in lighting Naive Solution: discarding the first few eigenfaces

7. 6 Pose Variation Basically, the existing solution can be divided into three types: multiple images in both training stage and recognition stage multiple images in training stage, but only one image in recognition stage single image based methods

8. 7 A Typical 3D FR System A4Vision Core Technology

9. 8 3D Facial Recognition Pipeline Features Point Clouds Depth Images Pattern Classifier 3D Face Detection Pre-proc. Face Normalization/Alignment Fine Alignment Noise Removal Hole Filling Smoothing Cropping Landmark Finding Coarse Alignment

10. 9 3D Face Detection This problem has not been touched so far! Simple heuristics such as nose tip In complex scenes, curvature analysis is generally used

11. 10 Pre-processing Artifact removal Noise removal: spikes (filters), clutter (manually), noise (median filter) Holes filling (Gaussian smoothing, linear interpolation, symmetrical interpolation)

12. 11 Face Normalization/Alignment Coarse alignment by Centre of mass, Plane fitted to the data Facial landmarks (eyes, nose tip) Fine alignment ICP (Iterative Conditional Proc.) Warping Elastic deformations

13. 12 3D Acquisition Systems Face specific Biometrics A4Vision Geometrix Modeling Cyberware Genex Inspeck Medeim Breuckmann

14. 13 3D Face Databases UND 275 subjects, 943 scans Shape + texture FRGC 400 subjects, 4007 scans Shape + texture 3D_RMA 120 subject, 6 scans Shape only GavabDB 61 subjects (9 scans) Shape only Pose, expression variations USF database 357 scans 3DFS generator Custom face databases 12 persons to ~6000 persons (A4Vision) UND GavabDB 3DFS

15. 14 3D Face Recognition Approaches Appearance-Based Methods Feature-Based Methods Model-Based Methods

16. 15 An Appearance-Based Method Illumination Cone Method

17. 16 Lambertian Model Lambertian shading assumes that the incoming light is reflected equally in all directions, without bias. The angle of incoming light has no effect on the direction in which it is reflected. Lambertian Phong

18. 17 Illumination cone For a Lambertian surface: Image x superpositioned with k light sources can be written as:

19. 18 Illumination cone Database images of Yale University: Different Illuminations Different Poses

20. 19 Illumination cone Image Acquisition

21. 20 Illumination cone Least Square estimation is used to find normal unit vectors.

22. 21 Illumination cone Illumination cone is a subspace covers the variation in illumination. Basis images Synthetic images Reconstructed surface by means of GBR Ambiguity

23. 22 Illumination cone 1.Representations and Algorithms for Face Recognition 2.Constructing 117 (19x7) different poses by means of planar transformations (non-linear warping) 3.Constructing the Illumination Cone of each pose from 80-120 different lighting conditions

24. 23 Illumination cone 4.Decreasing the number of lighting conditions using PCA dimension reduction down to 11 5.Applying SVD based methods to reduce the number of database images (11x117) down to 100 6.These 100 images form the image basis space for each person

25. 24 A Feature-Based Method Structured-Light Pattern Method

26. 25 Structured-Light Surface Rendering

27. 26 Structured-Light Surface Rendering Striped images Reconstructed surface

28. 27 Structured-Light Surface Rendering Curvature Analysis for Surface Matching

29. 28 Triangulation-Interpolation

30. 29 A Model-Based Method Elastic Bunch Graph Matching

31. 30 Elastic Bunch Graph use Gabor wavelet transform to extract face features so that the recognition performance can be invariant to the variation in poses.

32. 31 Elastic Bunch Graph Gabor wavelet decomposition Gabor kernels

33. 32 Gabor Filters

34. 33 Jets Small patch gray values Wavelet transform

35. 34 Comparing Jets Amplitude similarity Phase similarity

36. 35 Comparing Jets

37. 36 Face Bunch Graphs (FBG) Stack like general representation Graph similarity function

38. 37 Graph Extraction Step 1: find approximate face position Step 2: refine position and size Step 3: refine size and find aspect ratio Step 4: local distortion

39. 38 Recognition Comparing image graph Recognized for highest similarity

40. 39 Open Issues & Challenges Uncontrolled acquisition Non-cooperative Different lighting conditions Texture map + shape map inconsistencies Real-time 3D video data Computational complexity Issues related to performance assessment Publicly available standard face databases Quality (resolution) of the data Artifacts such as eyeglasses

41. 40 Conclusions 3D face recognition systems were proposed to overcome expression, illumination, and pose challenges Illumination correction is simpler Facial landmark localization is better The core algorithm, ICP, has limited capabilities Not suitable for non-rigid deformations