1. IRCDL'07 - Padova, January 2007 3D Face Recognition using iso-Geodesic Surfaces Stefano Berretti, Alberto Del Bimbo, Pietro Pala Dipartimento di Sistemi e Informatica University of Firenze, Firenze, ITALY {berretti,delbimbo,pala}@dsi.unifi.it

2. Motivation  Detection and identification of human faces have been largely addressed mainly focussing on 2D still images and videos.  Recently, the increasing availability of three-dimensional (3D) data, has renewed the interest in 3D face models to improve the effectiveness of face recognition systems.  Solutions based on 3D face models are expected to feature less sensitivity to lighting conditions and pose variations.  3D face models are acquired through laser scanners, structured ligth scanners or multi camera systems, in the form of range images, cloud of points or polygonal meshes.  Usually texture data is not available.

3. 3D face recognition approaches  2D to 2D (using deformations of a 3D model):  The input is 2D image; the database contains 2D images. Recognition is performed by comparing the deformations of a 3D face model over the database images and the query image (using morphing) [Blanz, TPAMI03].  3D to 2D:  3D to 2D transformation. Techniques for 2D image matching (e.g., PCA, Gabor filters, wavelets, etc.) are used to perform recognition in 2D [Pan, CVPR’05], [Wang, CVPR06].  Model based solutions. A 3D annotated face model is deformed to fit a 3D query model. The deformed model is projected onto a 2D image. Recognition is performed by comparing 2D projected images [Jain, CVPR06], [Kakadiaris, BMVC06].  2D+3D:  3D face queries are matched with 3D face models; 2D face images are matched with 2D face image models, separately. Results of the two matchings are combined to achieve better recognition accuracy [Beumier, PRL01], [Wang, IVC05].  3D to 3D (point clouds):  3D face rigid surfaces acquired as point clouds are compared using the Iterative Closest Point (ICP) algorithm [Mian, BMVC05], [Chang, TPAMI06]. ......  Current research challenges:  Recognition in the presence of expression variations.  Matching efficiency.

4. Proposed solution  We propose a 3D to 3D face recognition approach in which the structural information of a 3D face model, given as a mesh, is directly captured in the 3D space by modeling the basic 3D shape and the relative arrangement of iso-geodesic stripes identified on the model surface.  Three steps:  Extraction of iso-geodesic stripes according to the geodesic distances from fiducial points.  Modeling of the shape and 3D spatial relationships between stripes (surfaces in the 3D space).  Construction and matching of a graph based face model.

5. Geodesic distance from fiducial points  Iso-geodesic stripes are defined with reference to a function f defined on the face surface.  Value of f at a generic vertex v is defined as the geodesic distance  (v,u) between v and a fiducial vertex u (the nose tip).  Due to the polygonal approximation of the model surface, the geodesic distance  is computed as the shortest path on the edges of the mesh by using the Dijkstra’s algorithm.  For regular meshes, this provides a fair approximation of the real geodesic distance.  Normalized values of the distance are obtained dividing the geodesic distance by the Euclidean eye-to-nose distance.  This guarantees invariance w.r.t. scaling and pose of the face.  Since the eye-to-nose distance is invariant to face expressions, this normalization does not bias values of  under expression changes. Euclidean distance Geodesic distance v u

6.  The nose tip and the eyes cavities are identified using mean and Gaussian curvatures according to state of the art solutions*.  Models are aligned with the nose tip pointing along the z axis.  Distances from the nose tip are quantized into n intervals c 1,..., c n. Accordingly, n level set stripes are identified on the model surface, the i-th stripe corresponding to the set of surface points on which the value of the distance falls in the limits of interval c i.  Stripes are surfaces in the 3D space.  Information captured by the stripes is modeled by their 3D mutual spatial relationships using the weighted walkthroughs approach. Stripes extraction (*) K.I. Chang, K.W. Bowyer, P.J. Flynn. “Multiple Nose Region Matching for 3D Face Recognition Under Varying Facial Expression,” IEEE Trans. on Pattern Analysis and Machine Intelligence, October 2006.

7.  In a 3D reference system, with coordinate axes X, Y, Z, the projections of two points a = and b = on each axis, can take three different orders: before, coincident, after.  The combination of the three projections results in 27 different 3D displacements which can be encoded by a triple of indexes :  In general, points of 3D extended sets A, B can be connected through multiple different displacements.  The triple, is a walkthrough from A to B if it encodes the displacement between at least one pair of points a  A, b  B. Modeling 3D spatial relationships x a b z y zaza xaxa yaya zbzb xbxb ybyb i=+1 k=+1 j=-1 z y j A B x k i

8. 3D weighted walkthroughs (WW)  Relevance of each walkthrough is measured by a weight w i,j,k (A,B) evaluating the number of pairs of points a  A, b  B, whose displacement is captured by the direction.  Formally, weights w i,j,k (A,B) are evaluated as integral measures over the six-dimensional set of point pairs in A and B: K ijk acts as dimensional normalization factor; C ±1 (.) are the characteristic functions of the positive and negative real semi-axis (0,+oo) and (-oo,0), respectively. C 0 (.)= δ(.) denotes the Dirac's function and acts as the characteristic function of the singleton set {0}.  Weights between A and B are organized in a 3x3x3 matrix w(A,B), of indexes i, j, k. W 011 W -111 W 111 W 001 W -101 W 101 W 0-11 W -1-11 W 1-11 +1 j 0 -1 0 +1 i W 010 W -110 W 110 W 100 W 1-10 W 01-1 W -11-1 W 11-1 W 10-1 W 1-1-1 +1 0k0k w(A,B)

9. Compositional computation  Integral properties ensure that the 27 weights are reflexive (w i,j,k (A,B) = w -i,-j,-k (B,A)), and invariant with respect to shifting and scaling. In addition, they are compositional:  According to this, any complex entity is decomposed in a set of 3D rectangular voxels of a uniform partitioning of the space.  The integral computation is cast to the linear combination of a set of sub-integrals computed in closed form on rectangular domains arranged in a fixed number of possible different spatial positions. 0 0 0 w k=1 (a n,b m ) = 0 0 0 ¼ ½ ¼ 0 0 0 w k=0 (a n,b m ) = 0 0 0 ½ 1 ½ 0 0 0 w k=-1 (a n,b m ) = 0 0 0 ¼ ½ ¼ z y anan x bmbm anan bmbm

10. WW between surfaces (stripes) in 3D  Computation of 3D spatial relationships between surfaces directly descends from the general case.  Volumetric integrals become surface integrals extended to the points of two surfaces.  Numerical computation of the surface integrals is avoided by exploiting the property of compositionality, and regarding surface entites as volumetric entities with infinitesimal thickness.  Linear combination of integrals computed in closed form between elementary rectangular voxels of a uniform partitioning of the space.

11.  Given the w(A,B) matrix, measuring the relationship between entities A and B, three directional weights, taking values in [0,1], are computed on the eight corner weights:  These account for the degree by which B is on the right ( ), up ( ) and in front of A ( ).  Similarly, three weights account for the alignment along each of the three reference directions of the space:  Finally, three weights account for the joint alignment along two directions of the space: Directional and alignment weights w(A,B) 3x3x3 matrix w -111 w 111 w -1-11 w 1-11 w -111 w 111 w -11-1 w 11-1 w 111 w 1-11 w 11-1 w 1-1-1

12. Distance between WW  Dis-similarity in the arrangement of pairs of entities and captured by matrices w(A,B)=w and w(A’,B’)=w’, is evaluated by a distance D(w,w') which combines the differences between homologous coefficients in the space of 27-tuples of WW.  Using the directional and alignment weights, this is expressed as: where are non negative numbers with sum equal to 1.

13. Graph representation  WW are computed for every pair of stripes, including the pair composed by a stripe and itself.  A face with N stripes is represented by N*(N+1)/2 relationship matrixes. This model is cast to a graph representation by regarding face stripes as graph nodes and their mutual spatial relationships as graph edges:   assigns to a node the WW computed between the node and itself;  associates to every edge the WW computed between the two nodes the edge connects to.

14. Efficient matching  Due to the normalization of geodesic distances, homologous stripes in different models capture the same distance interval.  Face matching is reduced to the comparison between homologous iso-geodesic stripes and their relationships: An interpretation Γ associates homologous nodes in the two representations, that is Γ(p k ) = r k, k=1,...., N, being p k and r k nodes of the template and reference face graphs: t1t1 t2t2 t3t3 t4t4 t5t5 t6t6 t7t7 r1r1 r2r2 r3r3 r4r4 r5r5 r6r6 r7r7

15. 3D face database  Face recognition experiments have been conducted on the 3D facial models of the GavabDB database (publicly available at http://gavab.escet.urjc.es). http://gavab.escet.urjc.es  The database includes 61 people (45 male and 16 female).  They are Caucasian and most of them are aged between 18 and 40.  For each person, 7 different models are taken differing in pose or facial expression, resulting in 427 facial models.  2 frontal and 2 rotated models with neutral facial expression.  3 frontal models in which the person laughs, smiles or exhibits a random gesture.

16. Recognition: neutral expression  In the recognition experiments, for each person one of the two frontal models with neutral expression is used as reference (gallery) model.  A gallery of 61 neutral models is obtained.  The remaining 366 models are used as templates for the recognition (probes).  A stripe extent of 0.08 is used in the experiments. Dynamic of the distance using the first 8 stripes #12#8#4

17. Recognition: non-neutral expression  The proposed approach (IGS) provides a relatively high recognition accuracy also for variations in face expression.  This can be intuitively motivated by the fact that the geodesic distance between vertices of the mesh is almost invariant to moderate facial expression changes.  Large variability inside the given categories of face expressions.  In the “random” category, each subject is free to provide a generic facial expression.

18. Baseline comparison  Baseline comparison against the Iterative Closest Point (ICP) registration algorithm is performed.  ICP monotonically converges to a local minima with respect to the mean-square distance objective function computed between vertices of two models.  To avoid local minima in the ICP registration, the algorithm is initialized with the correspondence between fiducial points of the two models.

19. Current and future work  Facial stripes computed with respect to different/multiple fiducial points.  Improvement of recognition accuracy w.r.t. expression variations.  Experiments on larger and heterogeneous face databases.  Florida State University database (115 persons with 6 different expressions each).  The Face Recognition Grand Challenge database - FRGC v.2 (4007 3D scans of 466 persons, includes expression changes).  The BJUT-3D large scale chinese face database (500 models, neutral expression). Useful to study ethnic variation.  Comparison w.r.t. different 3D to 3D recognition approaches on common benchmark databases.