Berk Gökberk Boğaziçi University – Perceptual Intelligence Lab Turkey Principles of 3D Face Recognition

2 Promises and motivations 2D face recognition still requires help Pose, expression, illumination variations Possible solutions: other modalities? Video, infra-red, stereo, 3D Promises of 3D facial recognition High-security applications 3D shape information invariance Pose and illumination problems can be solved Better facial feature localization

3 Some scenarios 3D-to-3D 2D-to-2D via 3D 2D-to-3D or 3D-to-2D

4 How do we get 3D facial data? Stereo cameras Quality: low to medium Speed: fast Problems: reconstruction Structured-lights Quality: medium Speed: fast Problems: intrusive Laser scanners Quality: high Speed: slow Problems: intrusive, Shape from {shading, motion, video} N/A

5 3D Facial Recognition Pipeline Features Point Clouds Depth Images Pattern Classifier 3D Face Detection Pre-proc. Face Normalization/Alignment Fine Alignment Noise Removal Hole Filling Smoothing Cropping Landmark Finding Coarse Alignment

6 3D Face Detection This problem has not been touched so far! Simple heuristics such as nose tip In complex scenes, curvature analysis is generally used Ref: 3D face detection using curvature analysis, Alessandro Colombo, Claudio Cusano, Raimondo Schettini, Pattern Recognition 39 (2006) 444 – 455

7 Pre-processing Artifact removal Noise removal: spikes (filters), clutter (manually), noise (median filter) Holes filling (Gaussian smoothing, linear interpolation, symmetrical interpolation)

8 Face Normalization/Alignment Coarse alignment by Centre of mass, Plane fitted to the data Facial landmarks (eyes, nose tip) Fine alignment ICP Warping Elastic deformations

9 3D Facial Features

10 PC -ICP gives the distance -Hausdorff -Too many points

11 PC SN -Enhanced Gaussian Image -Too many normals -ICP gives the distance -Hausdorff -Too many points

12 PC SN PRO -Sparse -Easy to compare -Not fully descriptive -Enhanced Gaussian Image -Too many normals -ICP gives the distance -Hausdorff -Too many points

13 PC SN PRO CURV Mean Gaussian Shape Index Principal directions -Landmark detection -Segmentation -Sensitive to noise and the quality of data -Sparse -Easy to compare -Not fully descriptive -Enhanced Gaussian Image -Too many normals -ICP gives the distance -Hausdorff -Too many points

14 PC SN PRO CURV Mean Gaussian Shape Index Principal directions DI PCA -Landmark detection -Segmentation -Sensitive to noise and the quality of data -Sparse -Easy to compare -Not fully descriptive -Enhanced Gaussian Image -Too many normals -ICP gives the distance -Hausdorff -Too many points -Benefit from 2D literature -Easy to fuse with texture -Applicable to 2.5D only

15 PC SN PRO CURV Mean Gaussian Shape Index Principal directions DI TEX PCA Gabor -Landmark detection -Segmentation -Sensitive to noise and the quality of data -Sparse -Easy to compare -Not fully descriptive -Enhanced Gaussian Image -Too many normals -ICP gives the distance -Hausdorff -Too many points -Benefit from 2D literature -Easy to fuse with texture -Applicable to 2.5D only

16 Baseline algorithms When you design a new system, which algorithms should be selected to show your algorithm’s superiority? PCA Texture & Shape Match score for texture Match score for shape Use weighted sum Perform ICPOutput the surface matching error Face A Face B Baseline Algorithm 1 Baseline Algorithm 2

17 What are the scenarios tested? The discriminative power of texture and shape? Taken from: “Three-dimensional face recognition”, Bronstein, A.M., and Bronstein, M.M., and Kimmel, R. International Journal of Computer Vision 2005, Vol.64 No.1 p.5-30

18 What are the scenarios tested? The discriminative power of texture, shape, or texture+shape? Pose variations No disciplined analysis to compare 2D and 3D under pose variations Expression variations Most of the databases do not contain expression variations No comparison to 2D Illumination variations Image relighting Albedo estimation

19 Open Issues & Challenges Uncontrolled acquisition Non-cooperative Different lighting conditions Texture map + shape map inconsistenies Real-time 3D video data Computational complexity Issues related to performance assesment Publicly available standard face databases Quality (resolution) of the data? Artifacts such as eyeglasses Images taken from: A survey of approaches and challenges in 3D and multi-modal 3D + 2D face recognition, Kevin W. Bowyer, Kyong Chang, Patrick Flynn, Computer Vision and Image Understanding 101 (2006) 1–15

Quick notes on 3D acquisition systems & 3D face databases

21 3D Acquisition Systems Face specific Biometrics A4Vision Geometrix Modeling Cyberware Genex Inspeck Medeim Breuckmann General sensors Minolta V-910

22 3D Face Databases UND 275 subjects, 943 scans Shape + texture FRGC 400 subjects, 4007 scans Shape + texture 3D_RMA 120 subject, 6 scans Shape only GavabDB 61 subjects (9 scans) Shape only Pose, expression variations USF database 357 scans 3DFS generator Custom face databases 12 persons to ~6000 persons (A4Vision) UNDUSF GavabDB 3DFS

23 Conclusions 3D face recognition systems were proposed to overcome expression, illumination, and pose challenges Illumination correction is simpler Facial landmark localization is better Baseline recognizers Combining depth maps with texture channel at the decision level The core algorithm, ICP, has limited capabilities Not suitable for non-rigid deformations Lots of research on shape channel representation Few in combining shape + texture

24 References Boğaziçi University Perceptual Intelligence Lab (PILAB) Signal and Image Processing Lab (BUSIM) Relevant Papers: Comparative analysis of decision-fusion methods for 3D face recognition B. Gökberk, L. Akarun, submitted for publication. Exact 2D-3D Facial Landmarking for Registration and Recognition Salah, A.A., H. Çınar, L. Akarun, B. Sankur, submitted for publication. 3D Shape-based Face Representation and Feature Extraction for Face Recognition B. Gökberk, M. O. İrfanoğlu, L. Akarun, Image and Vision Computing (accepted). 3D Face Recognition by Projection Based Methods H. Dutağacı, B. Sankur, Y. Yemez, in SPIE Conf. on Electronic Imaging, D/3D facial feature extraction Çınar Akakın, H., A.A. Salah, L. Akarun, B. Sankur, in SPIE Conf. on Electronic Imaging, Selection and Extraction of Patch Descriptors for 3D Face Recognition B. Gökberk, L. Akarun, ISCIS 2005, LNCS, Vol Springer 2005, pp D Face Recognition for Biometric Applications L. Akarun, B. Gökberk, A. A. Salah, EUSIPCO2005, Antalya, Turkey. Rank-based Decision Fusion for 3D Shape-based Face Recognition B. Gökberk, A. A. Salah, L. Akarun, AVBPA 2005, LNCS, Vol.3546 p D Shape-based Face Recognition using Automatically Registered Facial Surfaces M. O. İrfanoğlu, B. Gökberk, L. Akarun, ICPR2004, pp Contact: Berk Gökberk,

Additional Material

26 High Resolution 2D vs 2D 3D Shape+Texture 3D Texture Only 3D Shape Only Reference: Jonathon Phillips, FRGC Workshop, CVPR’05 Face Recognition Grand Challenge