1. 3D Face Modeling
Michaël De Smet

2. Topics to Discuss 3D Morphable Models 3D face reconstruction
Face recognition
Lip synchronization

3. Topics to Discuss 3D Morphable Models 3D face reconstruction
Face recognition
Lip synchronization

4. 3D Morphable Models Statistical model of shape and texture
Derived from laser scans
USF DARPA HumanID 3D Face Database
Processing
Hole filling
Surface smoothing
Albedo estimation
Dense correspondence

5. 3D Morphable Models
-2s
+2s …
-2s
+2s …

6. Topics to Discuss 3D Morphable Models 3D face reconstruction
Face recognition
Lip synchronization

7. 3D Face Reconstruction
Fitting the 3DMM to one or more images of the same face
Scale
Rotation
Translation
Illumination ? ?
Shape
Texture
Optimization problem with > 100 parameters

8. 3D Face Reconstruction
Feature points
Feature alignment
Fitting result

9. 3D Face Reconstruction

10. Dealing with Occlusions

11. Dealing with Occlusions
Without occlusion
handling
With occlusion
handling

12. 3D Face Reconstruction

13. 3D Face Reconstruction

14. 3D Face Reconstruction

15. 3D Face Reconstruction

16. 3D Face Reconstruction

17. 3D Face Reconstruction

18. Topics to Discuss 3D Morphable Models 3D face reconstruction
Face recognition
Lip synchronization

19. Face Recognition ? ? Fit the 3DMM to an image of an unknown face Scale
Rotation
Translation
Illumination ? ?
Shape
Texture
Compare to database
Recognition

20. Face Recognition
In controlled settings, almost perfect recognition is possible
Pose 1
100.0%
Pose 2
Pose 3
N/A
Pose 4
Pose 5
95.7%
Training view

21. Face Recognition Uncontrolled environments are challenging
Face orientation unknown
Difficult illumination
Facial expressions
Occlusions
Low resolution
Motion blur …

22. Face Recognition
European parliament video: 21 persons, 86% correct

23. Face Recognition
VRT news broadcasts: 12 persons

24. Face Recognition VRT news broadcasts: 12 persons 82.3% correct

25. Topics to Discuss 3D Morphable Models 3D face reconstruction
Face recognition
Lip synchronization

26. Lip Synchronization Speech driven animation
Texture based, i.e. shape is fixed
Strategy:
Extract 3D model of speaker’s face
Track rigid motion of the face in video
Extract texture for each frame
Compute PCA model of texture
Train ANN to link phonemes and PCA coefficients (visemes)

27. System Overview Neural Network Face Synthesis Speech feature vectors
Automatic
Phone
Recognition
Neural
Network
Face
Synthesis
Speech
feature vectors
Facial
feature vectors

28. Training Setup Neural Network Training Face Analysis Speech
Automatic
Phone
Recognition
Neural
Network
Training
Face
Analysis
Speech
feature vectors
Facial
feature vectors

29. Video Processing 3D face model acquisition Rigid motion tracking
Normalized texture extraction
Texture feature extraction (PCA)

30. Video Processing
After tracking, a normalized texture map is extracted for each video frame
Pose and lighting invariant representation

31. Conclusion 3DMMs are a very powerful tool for face modeling
Many applications in computer vision and computer graphics