1. 1 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Expression-invariant representation of faces and its applications for 3D face recognition Michael M. Bronstein Department of Computer Science Technion – Israel Institute of Technology M.Sc. Seminar 2 November 2004

2. 2 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Biometrics in the age of patriarchs And Jacob went near unto Isaac his father; and he felt him, and said, ‘The voice is Jacob’s voice, but the hands are the hands of Esau’. And he recognized him not, because his hands were hairy, as his brother Esau’s hands. Genesis XXVII:22

3. 3 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Biometrics today IRIS FINGERPRINT PALM VOICE RETINA DNA SIGNATURE FACE

4. 4 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition ILLUMINATION + EXPRESSION ILLUMINATION + POSE ILLUMINATION + EXPRESSION + POSE A. Bronstein, M. Bronstein and R. Kimmel, “Expression-invariant 3D face recognition”, chapter in Face processing: advanced modeling and methods, Chellapa and Zhao (Eds.) to appear Problems of 2D face recognition

5. 5 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Approaches in 2D face recognition Few feature points can be reliably extracted Such features are usually insufficient for good recognition Appropriate model is a problem No good model for facial expressions INVARIANT REPRESENTATIONGENERATIVE APPROACH

6. 6 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Intrinsic problem of 2D face recognition A. Bronstein, M. Bronstein and R. Kimmel, “Expression-invariant 3D face recognition”, chapter in Face processing: advanced modeling and methods, Chellapa and Zhao (Eds.) to appear

7. 7 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition 2D vs 3D in face recognition Simple acquisition – legacy hardware and databases Sensitive to everything (lighting, pose, makeup, expressions) Insensitive to lighting, pose, make up Requires special hardware Sensitive to expressions Sensitive to aging and plastic surgery (?)

8. 8 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Face is a smooth, connected, compact 2D Riemannian manifold Parametrization Metric tensor Riemannian geometry basics Geodesic distances

9. 9 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition ISOMETRICNON-ISOMETRIC Isometry is a transformation that preserves geodesic distances Isometric model A. Bronstein, M. Bronstein and R. Kimmel, “Expression-invariant 3D face recognition”, chapter in Face processing: advanced modeling and methods, Chellapa and Zhao (Eds.) to appear Isometric model: Facial expressions = isometries of some initial facial surface (neutral expression)

10. 10 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition The open mouth problem WITHOUTH TOPOLOGICAL CONSTRIANT Open mouth is not an isometry Isometric model is true for expressions with closed or open mouth Extension: topologically-constrained isometric model WITH TOPOLOGICAL CONSTRIANT M. Bronstein, A. Bronstein and R. Kimmel, “Expression-invariant representation for human faces”, submitted

11. 11 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition RIEMANNIANEUCLIDEAN Geodesic distances are an invariant description of the surface… …but are inconvenient to work with A. Elad and R. Kimmel, CVPR 2001 A. Elad and R. Kimmel, IEEE Trans. PAMI, 2003 [Elad & Kimmel 2001]: embed isometric surfaces into a low- dimensional Euclidean space and treat them as rigid objects Canonical forms

12. 12 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition - ( m > 2) -dimensional manifold (embedding space) Isometric embedding - N  N matrix of original geodesic distances - N  N matrix of distances in the embedding space A mapping between finite metric spaces such that

13. 13 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Embedding problem in cartography A. Bronstein, M. Bronstein and R. Kimmel, “Three-dimensional face recognition”, submitted to IJCV The globe cannot be embedded into a plane according to Gauss Theorema Egregium. GLOBE (HEMISPHERE)PLANAR MAP

14. 14 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Does an isometric embedding always exist ? N. Linial Example of 4 points on a sphere that cannot be isometrically embedded into an Euclidean space of ay finite dimension. 4 POINTS ON A SPHEREA NEAR-ISOMETRIC EMBEDDING

15. 15 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition - N  m matrix of canonical form parametric coordinates Embedding error RAW STRESS: NORMALIZED STRESS: I. Borg and P. Grönen, Modern multidimensional scaling, Springer, 1997

16. 16 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Multidimensional scaling Nm optimization variables I. Borg and P. Grönen, Modern multidimensional scaling, Springer, 1997 Multidimensional scaling (MDS) problem: Optimum defined up to an isometry in Non-convex optimization problem Optimum = canonical form

17. 17 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Euclidean embedding – LS MDS where Choose I. Borg and P. Grönen, Modern multidimensional scaling, Springer, 1997 For require Use

18. 18 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Alignment Eliminate first-order moments Reorder the axes Eliminate mixed second-order moments Force the sign where

19. 19 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Canonical forms of faces (closed mouth) FACIAL SURFACES CANONICAL FORMS A. Bronstein, M. Bronstein and R. Kimmel, “Three-dimensional face recognition”, submitted to IJCV

20. 20 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition FACIAL SURFACES CANONICAL FORMS Canonical forms of faces (open mouth) M. Bronstein, A. Bronstein and R. Kimmel, “Expression-invariant representation for human faces”, submitted

21. 21 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Spherical embedding A. Bronstein, M. Bronstein and R. Kimmel, “On isometric embedding of facial surfaces into S 3 ”, subm. ScaleSpace Choose as a sphere immersed into Geodesic distances (arcs of great circles) Minimize the normalized stress w.r.t. the parametric coordinatesparametric coordinates Alignment performed using Euclidean moments in

22. 22 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Spherical embedding example Maximum-variance projection onto R 3 of a facial surface embedded into S 3 with different radii R = 5 cm A. Bronstein, M. Bronstein and R. Kimmel, “On isometric embedding of facial surfaces into S 3 ”, subm. ScaleSpace R = 7.5 cmR = 15 cm

23. 23 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Spherical embedding vs Euclidean embedding A. Bronstein, M. Bronstein and R. Kimmel, “On isometric embedding of facial surfaces into S 3 ”, subm. ScaleSpace SPHERE RADIUS R [cm] EMBEDDING ERROR

24. 24 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition 3DFACE system – hardware PROJECTOR CAMERA MONITOR CARD MOUNTING

25. 25 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition 3DFACE system – user interface

26. 26 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition 3DFACE system – pipeline

27. 27 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Isometric model validation 133 toothpaste markers placed on the face as invariant points Track how geodesic / Euclidean distances change due to expressions Lips cropped 16 different expressions M. Bronstein, A. Bronstein and R. Kimmel, “Expression-invariant representation for human faces”, submitted

28. 28 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Isometric model validation (cont.) ABSOLUTE CHANGE mmRELATIVE CHANGE % EUCLIDEAN GEODESIC M. Bronstein, A. Bronstein and R. Kimmel, “Expression-invariant representation for human faces”, submitted

29. 29 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Sensitivity to facial expressions A. Bronstein, M. Bronstein and R. Kimmel, “Three-dimensional face recognition”, submitted to IJCV M. Bronstein, A. Bronstein and R. Kimmel, “Expression-invariant representation for human faces”, submitted

30. 30 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Sensitivity to facial expressions (closed mouth) Visualization of the distances between faces in the facial expressions experiment. Each point represents a subject in the database. RIGID SURFACESCANONICAL FORMS A. Bronstein, M. Bronstein and R. Kimmel, “Three-dimensional face recognition”, submitted to IJCV

31. 31 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Sensitivity to facial expressions (open mouth) Visualization of the distances between faces in the facial expressions experiment. Each point represents a subject in the database. RIGID SURFACESTOPOL. CONSTR. CANONICAL FORMS M. Bronstein, A. Bronstein and R. Kimmel, “Expression-invariant representation for human faces”, submitted

32. 32 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Benchmarks A. Bronstein, M. Bronstein and R. Kimmel, “Three-dimensional face recognition”, submitted to IJCV Benchmark of face recognition algorithms on a database containing 220 instances of 30 faces with extreme facial expressions. RIGID CANONICAL EIGENFACES FALSE ACCEPTANCE RATE % FALSE REJECTION RATE % RECOGNITION RANK RECOGNITION ACCURACY %

33. 33 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Recognition example A. Bronstein, M. Bronstein and R. Kimmel, “Three-dimensional face recognition”, submitted to IJCV PROBEEIGENFACESRIGID SURFACECANONICAL FORM MORAN 129ORI 188SUSY 276MORAN 114 MICHAEL 17ALEX 40ALEX 39MICHAEL 2

34. 34 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Identical twins – find the differences The difference between Michael and Alex obtained by comparing the canonical forms reveals a slight difference in the geometry of their nose. MICHAELALEXDIFFERENCE MAP

35. 35 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Bibliography A. Elad and R. Kimmel, On bending invariant signatures for surfaces, Trans. IEEE PAMI 25(10): 1285-1295, 2003 A. Bronstein, M. Bronstein and R. Kimmel, Expression-invariant 3D face recognition, Proc. AVBPA 2003, LNCS 2688, 62-69, Springer A. Bronstein, M. Bronstein, R. Kimmel and A. Spira, Face recognition from facial surface metric, Proc. ECCV 2004, 225-237 A. Bronstein, M. Bronstein, R. Kimmel and E. Gordon, Fusion of 2D and 3D information in 3D face recognition, Proc. ICIP 2004 M. Bronstein, A. Bronstein and R. Kimmel, Three-dimensional face recognition, CIS Tech. Report 04, 2004, submitted to IJCV M. Bronstein, A. Bronstein and R. Kimmel, Expression-invariant representation of faces, submitted to PNAS A. Bronstein, M. Bronstein and R. Kimmel, On isometric embedding of facial surfaces into S 3, submitted to ScaleSpace

36. 36 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Isometric model for facial expressions NEUTRALAVERAGEMIN.MAX The uncertainty region around the face in the presence of facial expressions is so large that many other faces can fit in. A. Bronstein, M. Bronstein and R. Kimmel, “Expression-invariant 3D face recognition”, chapter in Face processing: advanced modeling and methods, Chellapa and Zhao (Eds.) to appear

37. 37 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Parametrization of S 3 A. Bronstein, M. Bronstein and R. Kimmel, “On isometric embedding of facial surfaces into S 3 ”, subm. ScaleSpace Geodesic distances