1 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Expression-invariant representation of faces and.

Slides:



Advertisements
Similar presentations
Coherent Laplacian 3D protrusion segmentation Oxford Brookes Vision Group Queen Mary, University of London, 11/12/2009 Fabio Cuzzolin.
Advertisements

Face Recognition. Introduction Why we are interested in face recognition? Why we are interested in face recognition? Passport control at terminals in.
Differential geometry I
1 Michael Bronstein Heat diffusion descriptors deformable Michael Bronstein Weizmann Institute of Science, 4 November 2010 Institute of Computational Science.
Ongoing Challenges in Face Recognition Peter Belhumeur Columbia University New York City.
Presented by: Mingyuan Zhou Duke University, ECE April 3, 2009
Topology-Invariant Similarity and Diffusion Geometry
1 Numerical Geometry of Non-Rigid Shapes Diffusion Geometry Diffusion geometry © Alexander & Michael Bronstein, © Michael Bronstein, 2010 tosca.cs.technion.ac.il/book.
Discrete Geometry Tutorial 2 1
University of Joensuu Dept. of Computer Science P.O. Box 111 FIN Joensuu Tel fax Isomap Algorithm.
Biometrics: Faces and Identity Verification in a Networked World
1 Numerical geometry of non-rigid shapes Geometry Numerical geometry of non-rigid shapes Shortest path problems Alexander Bronstein, Michael Bronstein,
1 Michael Bronstein Computational metric geometry Computational metric geometry Michael Bronstein Department of Computer Science Technion – Israel Institute.
Multidimensional scaling
Isometry invariant similarity
1 Michael Bronstein 3D face recognition Face recognition: New technologies, new challenges Michael M. Bronstein.
1 Numerical geometry of non-rigid shapes Lecture I – Introduction Numerical geometry of shapes Lecture I – Introduction non-rigid Michael Bronstein.
Spectral embedding Lecture 6 1 © Alexander & Michael Bronstein
Numerical geometry of objects
1 Bronstein 2 and Kimmel Extrinsic and intrinsic similarity of nonrigid shapes Michael M. Bronstein Department of Computer Science Technion – Israel Institute.
Department of Electrical and Computer Engineering Physical Biometrics Matthew Webb ECE 8741.
Correspondence & Symmetry
Uncalibrated Geometry & Stratification Sastry and Yang
1 Numerical geometry of non-rigid shapes Spectral Methods Tutorial. Spectral Methods Tutorial 6 © Maks Ovsjanikov tosca.cs.technion.ac.il/book Numerical.
1 Numerical geometry of non-rigid shapes Lecture II – Numerical Tools Numerical geometry of shapes Lecture II – Numerical Tools non-rigid Alex Bronstein.
Spectral Embedding Alexander Bronstein, Michael Bronstein
Differential geometry II
Numerical geometry of non-rigid shapes
1 Bronstein, Bronstein, and Kimmel Joint extrinsic and intrinsic similarity of non-rigid shapes Rock, paper, and scissors Joint extrinsic and intrinsic.
Non-Euclidean Embedding
Flattening via Multi- Dimensional Scaling Ron Kimmel Computer Science Department Geometric Image Processing Lab Technion-Israel.
Numerical geometry of non-rigid shapes
Paretian similarity for partial comparison of non-rigid objects
Comparison and Combination of Ear and Face Images in Appearance-Based Biometrics IEEE Trans on PAMI, VOL. 25, NO.9, 2003 Kyong Chang, Kevin W. Bowyer,
A Global Geometric Framework for Nonlinear Dimensionality Reduction Joshua B. Tenenbaum, Vin de Silva, John C. Langford Presented by Napat Triroj.
1 Bronstein 2 & Kimmel An isometric model for facial animation and beyond AMDO, Puerto de Andratx, 2006 An isometric model for facial animation and beyond.
Smart Traveller with Visual Translator for OCR and Face Recognition LYU0203 FYP.
1 Numerical Geometry of Non-Rigid Shapes Invariant shape similarity Invariant shape similarity © Alexander & Michael Bronstein, © Michael Bronstein,
1 Numerical geometry of non-rigid shapes Non-Euclidean Embedding Non-Euclidean Embedding Lecture 6 © Alexander & Michael Bronstein tosca.cs.technion.ac.il/book.
1 Numerical geometry of non-rigid shapes Expression-invariant face recognition Expression-invariant face recognition Lecture 8 © Alexander & Michael Bronstein.
1 Numerical geometry of non-rigid shapes Three-Dimensional Face Recognition Three Dimensional Face Recognition “And in stature he is small, chest broad,
1 Michael M. Bronstein Partial similarity of objects 17 December 2006 Partial similarity of objects, or how to compare a centaur to a horse Michael M.
1 Bronstein 2 & Kimmel Matching 2D articulated shapes using GMDS AMDO, Puerto de Andratx, 2006 Matching 2D articulated shapes using Generalized Multidimensional.
Three-Dimensional Face Recognition Using Surface Space Combinations Thomas Heseltine, Nick Pears, Jim Austin Advanced Computer Architecture Group Department.
NonLinear Dimensionality Reduction or Unfolding Manifolds Tennenbaum|Silva|Langford [Isomap] Roweis|Saul [Locally Linear Embedding] Presented by Vikas.
Lightseminar: Learned Representation in AI An Introduction to Locally Linear Embedding Lawrence K. Saul Sam T. Roweis presented by Chan-Su Lee.
1 M. Bronstein Multigrid multidimensional scaling Multigrid Multidimensional Scaling Michael M. Bronstein Department of Computer Science Technion – Israel.
1 Numerical geometry of non-rigid shapes Non-rigid correspondence Numerical geometry of non-rigid shapes Non-rigid correspondence Alexander Bronstein,
Biometrics: Ear Recognition
Karthiknathan Srinivasan Sanchit Aggarwal
Manifold learning: Locally Linear Embedding Jieping Ye Department of Computer Science and Engineering Arizona State University
Cosmology AA 2002/20031 The concept of Curvature Guido Chincarini This lecture is after the Luminosity Function and before the Metric_3D The idea is to.
Recognition Part II Ali Farhadi CSE 455.
Face Recognition and Feature Subspaces
Graph Embedding: A General Framework for Dimensionality Reduction Dong XU School of Computer Engineering Nanyang Technological University
Intelligent Database Systems Lab 國立雲林科技大學 National Yunlin University of Science and Technology Adaptive nonlinear manifolds and their applications to pattern.
Expression-invariant Face Recognition using Geodesic Distance Isometries Kerry Widder A Review of ‘Robust expression-invariant face recognition from partially.
1 Recognition by Appearance Appearance-based recognition is a competing paradigm to features and alignment. No features are extracted! Images are represented.
Biometrics Stephen Schmidt Brian Miller Devin Reid.
Computer Vision Lab. SNU Young Ki Baik Nonlinear Dimensionality Reduction Approach (ISOMAP, LLE)
Manifold learning: MDS and Isomap
Nonlinear Dimensionality Reduction Approach (ISOMAP)
3D Face Recognition Using Range Images
Facial Recognition Justin Kwong Megan Thompson Raymundo Vazquez-lugo.
1 Numerical geometry of non-rigid shapes Projects Quasi-isometries Project 1 © Alexander & Michael Bronstein tosca.cs.technion.ac.il/book Numerical geometry.
Intrinsic Data Geometry from a Training Set
Face Recognition and Feature Subspaces
Recognition: Face Recognition
Spectral Methods Tutorial 6 1 © Maks Ovsjanikov
NonLinear Dimensionality Reduction or Unfolding Manifolds
Presentation transcript:

1 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Expression-invariant representation of faces and its applications for 3D face recognition Michael M. Bronstein Department of Computer Science Technion – Israel Institute of Technology M.Sc. Seminar 2 November 2004

2 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Biometrics in the age of patriarchs And Jacob went near unto Isaac his father; and he felt him, and said, ‘The voice is Jacob’s voice, but the hands are the hands of Esau’. And he recognized him not, because his hands were hairy, as his brother Esau’s hands. Genesis XXVII:22

3 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Biometrics today IRIS FINGERPRINT PALM VOICE RETINA DNA SIGNATURE FACE

4 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition ILLUMINATION + EXPRESSION ILLUMINATION + POSE ILLUMINATION + EXPRESSION + POSE A. Bronstein, M. Bronstein and R. Kimmel, “Expression-invariant 3D face recognition”, chapter in Face processing: advanced modeling and methods, Chellapa and Zhao (Eds.) to appear Problems of 2D face recognition

5 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Approaches in 2D face recognition Few feature points can be reliably extracted Such features are usually insufficient for good recognition Appropriate model is a problem No good model for facial expressions INVARIANT REPRESENTATIONGENERATIVE APPROACH

6 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Intrinsic problem of 2D face recognition A. Bronstein, M. Bronstein and R. Kimmel, “Expression-invariant 3D face recognition”, chapter in Face processing: advanced modeling and methods, Chellapa and Zhao (Eds.) to appear

7 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition 2D vs 3D in face recognition Simple acquisition – legacy hardware and databases Sensitive to everything (lighting, pose, makeup, expressions) Insensitive to lighting, pose, make up Requires special hardware Sensitive to expressions Sensitive to aging and plastic surgery (?)

8 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Face is a smooth, connected, compact 2D Riemannian manifold Parametrization Metric tensor Riemannian geometry basics Geodesic distances

9 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition ISOMETRICNON-ISOMETRIC Isometry is a transformation that preserves geodesic distances Isometric model A. Bronstein, M. Bronstein and R. Kimmel, “Expression-invariant 3D face recognition”, chapter in Face processing: advanced modeling and methods, Chellapa and Zhao (Eds.) to appear Isometric model: Facial expressions = isometries of some initial facial surface (neutral expression)

10 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition The open mouth problem WITHOUTH TOPOLOGICAL CONSTRIANT Open mouth is not an isometry Isometric model is true for expressions with closed or open mouth Extension: topologically-constrained isometric model WITH TOPOLOGICAL CONSTRIANT M. Bronstein, A. Bronstein and R. Kimmel, “Expression-invariant representation for human faces”, submitted

11 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition RIEMANNIANEUCLIDEAN Geodesic distances are an invariant description of the surface… …but are inconvenient to work with A. Elad and R. Kimmel, CVPR 2001 A. Elad and R. Kimmel, IEEE Trans. PAMI, 2003 [Elad & Kimmel 2001]: embed isometric surfaces into a low- dimensional Euclidean space and treat them as rigid objects Canonical forms

12 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition - ( m > 2) -dimensional manifold (embedding space) Isometric embedding - N  N matrix of original geodesic distances - N  N matrix of distances in the embedding space A mapping between finite metric spaces such that

13 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Embedding problem in cartography A. Bronstein, M. Bronstein and R. Kimmel, “Three-dimensional face recognition”, submitted to IJCV The globe cannot be embedded into a plane according to Gauss Theorema Egregium. GLOBE (HEMISPHERE)PLANAR MAP

14 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Does an isometric embedding always exist ? N. Linial Example of 4 points on a sphere that cannot be isometrically embedded into an Euclidean space of ay finite dimension. 4 POINTS ON A SPHEREA NEAR-ISOMETRIC EMBEDDING

15 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition - N  m matrix of canonical form parametric coordinates Embedding error RAW STRESS: NORMALIZED STRESS: I. Borg and P. Grönen, Modern multidimensional scaling, Springer, 1997

16 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Multidimensional scaling Nm optimization variables I. Borg and P. Grönen, Modern multidimensional scaling, Springer, 1997 Multidimensional scaling (MDS) problem: Optimum defined up to an isometry in Non-convex optimization problem Optimum = canonical form

17 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Euclidean embedding – LS MDS where Choose I. Borg and P. Grönen, Modern multidimensional scaling, Springer, 1997 For require Use

18 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Alignment Eliminate first-order moments Reorder the axes Eliminate mixed second-order moments Force the sign where

19 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Canonical forms of faces (closed mouth) FACIAL SURFACES CANONICAL FORMS A. Bronstein, M. Bronstein and R. Kimmel, “Three-dimensional face recognition”, submitted to IJCV

20 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition FACIAL SURFACES CANONICAL FORMS Canonical forms of faces (open mouth) M. Bronstein, A. Bronstein and R. Kimmel, “Expression-invariant representation for human faces”, submitted

21 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Spherical embedding A. Bronstein, M. Bronstein and R. Kimmel, “On isometric embedding of facial surfaces into S 3 ”, subm. ScaleSpace Choose as a sphere immersed into Geodesic distances (arcs of great circles) Minimize the normalized stress w.r.t. the parametric coordinatesparametric coordinates Alignment performed using Euclidean moments in

22 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Spherical embedding example Maximum-variance projection onto R 3 of a facial surface embedded into S 3 with different radii R = 5 cm A. Bronstein, M. Bronstein and R. Kimmel, “On isometric embedding of facial surfaces into S 3 ”, subm. ScaleSpace R = 7.5 cmR = 15 cm

23 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Spherical embedding vs Euclidean embedding A. Bronstein, M. Bronstein and R. Kimmel, “On isometric embedding of facial surfaces into S 3 ”, subm. ScaleSpace SPHERE RADIUS R [cm] EMBEDDING ERROR

24 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition 3DFACE system – hardware PROJECTOR CAMERA MONITOR CARD MOUNTING

25 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition 3DFACE system – user interface

26 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition 3DFACE system – pipeline

27 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Isometric model validation 133 toothpaste markers placed on the face as invariant points Track how geodesic / Euclidean distances change due to expressions Lips cropped 16 different expressions M. Bronstein, A. Bronstein and R. Kimmel, “Expression-invariant representation for human faces”, submitted

28 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Isometric model validation (cont.) ABSOLUTE CHANGE mmRELATIVE CHANGE % EUCLIDEAN GEODESIC M. Bronstein, A. Bronstein and R. Kimmel, “Expression-invariant representation for human faces”, submitted

29 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Sensitivity to facial expressions A. Bronstein, M. Bronstein and R. Kimmel, “Three-dimensional face recognition”, submitted to IJCV M. Bronstein, A. Bronstein and R. Kimmel, “Expression-invariant representation for human faces”, submitted

30 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Sensitivity to facial expressions (closed mouth) Visualization of the distances between faces in the facial expressions experiment. Each point represents a subject in the database. RIGID SURFACESCANONICAL FORMS A. Bronstein, M. Bronstein and R. Kimmel, “Three-dimensional face recognition”, submitted to IJCV

31 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Sensitivity to facial expressions (open mouth) Visualization of the distances between faces in the facial expressions experiment. Each point represents a subject in the database. RIGID SURFACESTOPOL. CONSTR. CANONICAL FORMS M. Bronstein, A. Bronstein and R. Kimmel, “Expression-invariant representation for human faces”, submitted

32 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Benchmarks A. Bronstein, M. Bronstein and R. Kimmel, “Three-dimensional face recognition”, submitted to IJCV Benchmark of face recognition algorithms on a database containing 220 instances of 30 faces with extreme facial expressions. RIGID CANONICAL EIGENFACES FALSE ACCEPTANCE RATE % FALSE REJECTION RATE % RECOGNITION RANK RECOGNITION ACCURACY %

33 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Recognition example A. Bronstein, M. Bronstein and R. Kimmel, “Three-dimensional face recognition”, submitted to IJCV PROBEEIGENFACESRIGID SURFACECANONICAL FORM MORAN 129ORI 188SUSY 276MORAN 114 MICHAEL 17ALEX 40ALEX 39MICHAEL 2

34 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Identical twins – find the differences The difference between Michael and Alex obtained by comparing the canonical forms reveals a slight difference in the geometry of their nose. MICHAELALEXDIFFERENCE MAP

35 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Bibliography A. Elad and R. Kimmel, On bending invariant signatures for surfaces, Trans. IEEE PAMI 25(10): , 2003 A. Bronstein, M. Bronstein and R. Kimmel, Expression-invariant 3D face recognition, Proc. AVBPA 2003, LNCS 2688, 62-69, Springer A. Bronstein, M. Bronstein, R. Kimmel and A. Spira, Face recognition from facial surface metric, Proc. ECCV 2004, A. Bronstein, M. Bronstein, R. Kimmel and E. Gordon, Fusion of 2D and 3D information in 3D face recognition, Proc. ICIP 2004 M. Bronstein, A. Bronstein and R. Kimmel, Three-dimensional face recognition, CIS Tech. Report 04, 2004, submitted to IJCV M. Bronstein, A. Bronstein and R. Kimmel, Expression-invariant representation of faces, submitted to PNAS A. Bronstein, M. Bronstein and R. Kimmel, On isometric embedding of facial surfaces into S 3, submitted to ScaleSpace

36 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Isometric model for facial expressions NEUTRALAVERAGEMIN.MAX The uncertainty region around the face in the presence of facial expressions is so large that many other faces can fit in. A. Bronstein, M. Bronstein and R. Kimmel, “Expression-invariant 3D face recognition”, chapter in Face processing: advanced modeling and methods, Chellapa and Zhao (Eds.) to appear

37 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Parametrization of S 3 A. Bronstein, M. Bronstein and R. Kimmel, “On isometric embedding of facial surfaces into S 3 ”, subm. ScaleSpace Geodesic distances