18/12/2006 The University of York 1 A Literature Review of Image- based Face Recognition Quan Ju PhD student Department of Computer Science The University of York

18/12/2006The University of York2 Background Introduction What is Face Recognition? What is Face Recognition? Face Recognition is a popular application of computer vision in recent years. Face Recognition is a popular application of computer vision in recent years. Not only the computer science researchers, but also the psychologists and neuroscientists are involved in this area. Not only the computer science researchers, but also the psychologists and neuroscientists are involved in this area. Why do we need Face Recognition? Why do we need Face Recognition? Strong need for personal identification and recognition without the cooperation of the participants. Strong need for personal identification and recognition without the cooperation of the participants. Commercial, security and law applications require the use of face recognition technology. Commercial, security and law applications require the use of face recognition technology.

18/12/2006The University of York3 Face Recognition Scenarios Face Verification: Face Verification: Is this person who he says he is? One to one matching process. Face Identification: Face Identification: Who is this person? One to many matching process Watch list: Watch list: Who are you looking for?

18/12/2006The University of York4 Difficulties in Face Recognition Head pose Head pose Illumination Illumination Facial expression Facial expression Hair Hair Aging problem Aging problem Occlusion e.g. glasses, scarf etc. Occlusion e.g. glasses, scarf etc.

18/12/2006The University of York5 Image-based Face recognition approaches Appearance-based face recognition Appearance-based face recognition Linear Analysis: PCA,ICA,LDA Non-linear analysis Model-based face recognition Model-based face recognition Elastic Bunch Graph Matching 2D Morphable Model 3D Morphable Model

18/12/2006The University of York6 Linear Analysis Classical linear appearance-based analysis - PCA, ICA and LDA each has its own basis vectors of a high dimensional face image space. Classical linear appearance-based analysis - PCA, ICA and LDA each has its own basis vectors of a high dimensional face image space. By using those linear analysis method, the face vectors can be projected to the basis vectors. By using those linear analysis method, the face vectors can be projected to the basis vectors. Through the projecting from a higher dimensional input image space to a lower dimensional space, dimensionality of original input image space is reduced. Through the projecting from a higher dimensional input image space to a lower dimensional space, dimensionality of original input image space is reduced. The matching score between the test face image and training images can be achieved by calculation the differences between their projection vectors. The higher the score, the more similar between these two face images. The matching score between the test face image and training images can be achieved by calculation the differences between their projection vectors. The higher the score, the more similar between these two face images.

18/12/2006The University of York7 Image Space Image vector and image subspace Image vector and image subspace x 1 represents a p×q image; x is a matrix of image vectors. Above is three 1x2 pixel image examples. Similar images locate close together, otherwise they are away from each other.

18/12/2006The University of York8 Principal Component Analysis The main idea of the principal component analysis is to find the vectors which best describe the distribution of face images within the entire image space. The main idea of the principal component analysis is to find the vectors which best describe the distribution of face images within the entire image space. PCA is an orthogonal transformation of the coordinate system in which the pixels are described. The new coordinate values are principal component PCA is an orthogonal transformation of the coordinate system in which the pixels are described. The new coordinate values are principal component Face space is comprised of eigenfaces, which are the eigenvectors of the set of the faces. Face space is comprised of eigenfaces, which are the eigenvectors of the set of the faces. PCA is performed by projecting a new image into the subspace called face space spanned by the eigenfaces and then classifying the face by comparing its position in face space with the positions of known individuals. PCA is performed by projecting a new image into the subspace called face space spanned by the eigenfaces and then classifying the face by comparing its position in face space with the positions of known individuals. PCA aims to extract a subspace where the variance is maximized PCA aims to extract a subspace where the variance is maximized

18/12/2006The University of York9 Independent Component Analysis PCA derives only the most expressive features which are unrelated to actual face recognition, and in order to improve performance additional discriminant analysis is needed. PCA derives only the most expressive features which are unrelated to actual face recognition, and in order to improve performance additional discriminant analysis is needed. ICA provide a more powerful data representation than PCA as its aim is to provide an independent rather than uncorrelated image decomposition and representation. ICA provide a more powerful data representation than PCA as its aim is to provide an independent rather than uncorrelated image decomposition and representation. ICA is a generalization of PCA. ICA is a generalization of PCA.

18/12/2006The University of York10 Linear Discriminant Analysis Similar images projections are close together, different images projections locate far away when using PCA, but the projection from different classes of images are mixed together. Similar images projections are close together, different images projections locate far away when using PCA, but the projection from different classes of images are mixed together. LDA is also called Fisher Discriminant Analysis LDA is also called Fisher Discriminant Analysis LDA is able to maximize the ratio of between-class distribution to that of within-class distribution. LDA is able to maximize the ratio of between-class distribution to that of within-class distribution.

18/12/2006The University of York11 Nonlinear analysis Linear discriminant methods are insensitive to the relationship among multiple pixels in the images. Some nonlinear relations may exist in a face image, especially under a complicated variation in viewpoint, illumination and facial expression which is highly nonlinear. Linear discriminant methods are insensitive to the relationship among multiple pixels in the images. Some nonlinear relations may exist in a face image, especially under a complicated variation in viewpoint, illumination and facial expression which is highly nonlinear. To extract nonlinear features of images, Linear analysis method was extended to nonlinear analysis such as Kernel PCA, Kernel ICA and Kernel FLD etc. To extract nonlinear features of images, Linear analysis method was extended to nonlinear analysis such as Kernel PCA, Kernel ICA and Kernel FLD etc. By using nonlinear analysis approaches the original input image space is projected nonlinearly onto a high dimensional feature space. In this high dimensional space, the distribution of image vectors could be simplified to linear pattern. By using nonlinear analysis approaches the original input image space is projected nonlinearly onto a high dimensional feature space. In this high dimensional space, the distribution of image vectors could be simplified to linear pattern.

18/12/2006The University of York12 Model-based face recognition The model-based face recognition scheme is aimed at constructing a model of the human face, which is able to capture the facial variations. The model-based face recognition scheme is aimed at constructing a model of the human face, which is able to capture the facial variations. Model-based approaches derive distance and relative position features from the placement of internal facial elements (eyes, nose). Model-based approaches derive distance and relative position features from the placement of internal facial elements (eyes, nose). Generally, a face model contains the information of shape and texture of the face. Generally, a face model contains the information of shape and texture of the face.

18/12/2006The University of York13 Bunch Graph Human faces have a similar topological structure. Human faces have a similar topological structure. Face can be structured by nodes located at some specific points and edges labeled with distance vectors, then a face graph is produced. Face can be structured by nodes located at some specific points and edges labeled with distance vectors, then a face graph is produced. Face Bunch graph is generated from a set of sample face images. The FBG serves as a general representation of a set of faces. Face Bunch graph is generated from a set of sample face images. The FBG serves as a general representation of a set of faces. The stacks of discs on a node contain a bunch of description of facial features. The stacks of discs on a node contain a bunch of description of facial features. Each stack of discs called a jet represents an alternative of facial feature description. Each stack of discs called a jet represents an alternative of facial feature description. The edges are labeled with averages of distance vectors. The edges are labeled with averages of distance vectors.

18/12/2006The University of York14 Elastic Bunch Graph Matching To recognize a new face by elastic bunch graph matching, the fiducial points are positioned so as to extract a graph, which maximize a graph similarity between this graph and the FBG. To recognize a new face by elastic bunch graph matching, the fiducial points are positioned so as to extract a graph, which maximize a graph similarity between this graph and the FBG. After the nodes has been located on the new face, the face can be recognized by comparing the similarity between that the graph of this face and graphs of every face store in the FBG. After the nodes has been located on the new face, the face can be recognized by comparing the similarity between that the graph of this face and graphs of every face store in the FBG.

18/12/2006The University of York15 An Active Appearance Model The AAM is constructed based on a set of labeled images, where landmark points are marked on each example face at key positions to describe the facial features. The AAM is constructed based on a set of labeled images, where landmark points are marked on each example face at key positions to describe the facial features. Models are combined together by using Linear Analysis methods such as PCA. Models are combined together by using Linear Analysis methods such as PCA. The vector of parameters for the combined model is controlling the shape and texture of models. The vector of parameters for the combined model is controlling the shape and texture of models. AAM fitting is applied to seek a set of model parameters that best represents the test face image. AAM fitting is applied to seek a set of model parameters that best represents the test face image. The goal of recognition is to find the best match between the test parameter vector and training parameter vector. The goal of recognition is to find the best match between the test parameter vector and training parameter vector.

18/12/2006The University of York16 3D Morphable model Human face is a surface lying in the 3D space. Thus, the 3D model is more suitable for representing faces, Human face is a surface lying in the 3D space. Thus, the 3D model is more suitable for representing faces, 3D model has stronger ability to minimize the problems of head pose, illumination. 3D model has stronger ability to minimize the problems of head pose, illumination. 3D morphable model is extended from 2D morphable model - AAM. 3D morphable model is extended from 2D morphable model - AAM. Similar recognition methods on 2D morphable model can be improved and applied on 3D model as well. Similar recognition methods on 2D morphable model can be improved and applied on 3D model as well.

18/12/2006The University of York17 Face Databases and Performance Evaluation There are about 28 face databases available currently, such as FERET, XM2VTS and UMIST etc. There are about 28 face databases available currently, such as FERET, XM2VTS and UMIST etc. How to choose the suitable database based on the task given and the algorithm needs. How to choose the suitable database based on the task given and the algorithm needs. FERET is a poplar face image database, which contains 1564 sets of images for a total of 14,126 images that includes 1199 individuals and 365 duplicate sets of images. FERET is a poplar face image database, which contains 1564 sets of images for a total of 14,126 images that includes 1199 individuals and 365 duplicate sets of images. False Acceptance / False Rejection and Equal Error Rate are scores to evaluate the similarity between a test pattern and a template. False Acceptance / False Rejection and Equal Error Rate are scores to evaluate the similarity between a test pattern and a template. The Face Recognition Vendor Test was started from 2000 based on the FERET database. The database used in FRVT was extended 2 years later in FRVT2002. The Face Recognition Vendor Test was started from 2000 based on the FERET database. The database used in FRVT was extended 2 years later in FRVT2002.

18/12/2006The University of York18 References W. Zhao, R. Chellappa, A. Rosenfeld, P.J. Phillips, Face Recognition: A Literature Survey, ACM Computing Surveys, 2003, pp W. Zhao, R. Chellappa, A. Rosenfeld, P.J. Phillips, Face Recognition: A Literature Survey, ACM Computing Surveys, 2003, pp X. Lu, Image Analysis for Face Recognition, personal notes, May 2003, 36 pages X. Lu, Image Analysis for Face Recognition, personal notes, May 2003, 36 pages H. Moon, P.J. Phillips, Computational and Performance aspects of PCA-based Face Recognition Algorithms, Perception, Vol. 30, 2001, pp M.A. Turk, A.P. Pentland, Face Recognition Using Eigenfaces, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3-6 June 1991, Maui, Hawaii, USA, pp H. Moon, P.J. Phillips, Computational and Performance aspects of PCA-based Face Recognition Algorithms, Perception, Vol. 30, 2001, pp M.A. Turk, A.P. Pentland, Face Recognition Using Eigenfaces, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3-6 June 1991, Maui, Hawaii, USA, pp M.A. Turk, A.P. Pentland, Face Recognition Using Eigenfaces, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3-6 June 1991, Maui, Hawaii, USA, pp M.A. Turk, A.P. Pentland, Face Recognition Using Eigenfaces, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3-6 June 1991, Maui, Hawaii, USA, pp C. Liu, H. Wechsler, Comparative Assessment of Independent Component Analysis (ICA) for Face Recognition, Proc. of the Second International Conference on Audio- and Video-based Biometric Person Authentication, AVBPA'99, March 1999, Washington D.C., USA, pp C. Liu, H. Wechsler, Comparative Assessment of Independent Component Analysis (ICA) for Face Recognition, Proc. of the Second International Conference on Audio- and Video-based Biometric Person Authentication, AVBPA'99, March 1999, Washington D.C., USA, pp A. Pentland, B. Moghaddam, T. Starner, View-Based and Modular Eigenspaces for Face Recognition, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, June 1994, Seattle, Washington, USA, pp A. Pentland, B. Moghaddam, T. Starner, View-Based and Modular Eigenspaces for Face Recognition, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, June 1994, Seattle, Washington, USA, pp K. Etemad, R. Chellappa, Discriminant Analysis for Recognition of Human Face Images, Journal of the Optical Society of America A, Vol. 14, No. 8, August 1997, pp K. Etemad, R. Chellappa, Discriminant Analysis for Recognition of Human Face Images, Journal of the Optical Society of America A, Vol. 14, No. 8, August 1997, pp W. Zhao, R. Chellappa, A. Krishnaswamy, Discriminant Analysis of Principal Components for Face Recognition, Proc. of the 3rd IEEE International Conference on Face and Gesture Recognition, FG'98, April 1998, Nara, Japan, p. 336 W. Zhao, R. Chellappa, A. Krishnaswamy, Discriminant Analysis of Principal Components for Face Recognition, Proc. of the 3rd IEEE International Conference on Face and Gesture Recognition, FG'98, April 1998, Nara, Japan, p. 336 A.M. Martinez, A.C. Kak, PCA versus LDA, IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 23, No. 2, 2001, pp A.M. Martinez, A.C. Kak, PCA versus LDA, IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 23, No. 2, 2001, pp J. Lu, K.N. Plataniotis, A.N. Venetsanopoulos, Face Recognition Using LDA-Based Algorithms, IEEE Trans. on Neural Networks, Vol. 14, No. 1, January 2003, pp J. Lu, K.N. Plataniotis, A.N. Venetsanopoulos, Face Recognition Using LDA-Based Algorithms, IEEE Trans. on Neural Networks, Vol. 14, No. 1, January 2003, pp C. Liu, H. Wechsler, Evolutionary Pursuit and Its Application to Face Recognition, IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 22, No. 6, June 2000, pp C. Liu, H. Wechsler, Evolutionary Pursuit and Its Application to Face Recognition, IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 22, No. 6, June 2000, pp L. Wiskott, J.-M. Fellous, N. Krueuger, C. von der Malsburg, Face Recognition by Elastic Bunch Graph Matching, Chapter 11 in Intelligent Biometric Techniques in Fingerprint and Face Recognition, eds. L.C. Jain et al., CRC Press, 1999, pp L. Wiskott, J.-M. Fellous, N. Krueuger, C. von der Malsburg, Face Recognition by Elastic Bunch Graph Matching, Chapter 11 in Intelligent Biometric Techniques in Fingerprint and Face Recognition, eds. L.C. Jain et al., CRC Press, 1999, pp M.-H. Yang, Kernel Eigenfaces vs. Kernel Fisherfaces: Face Recognition Using Kernel Methods, Proc. of the Fifth IEEE International Conference on Automatic Face and Gesture Recognition, May 2002, Washington D.C., USA, pp M.-H. Yang, Kernel Eigenfaces vs. Kernel Fisherfaces: Face Recognition Using Kernel Methods, Proc. of the Fifth IEEE International Conference on Automatic Face and Gesture Recognition, May 2002, Washington D.C., USA, pp H. Yang, Face Recognition Using Kernel Methods, Advances in Neural Information Processing Systems, T. Diederich, S. Becker, Z. Ghahramani, Eds., 2002, vol. 14, 8 pages.-H. Yang, Face Recognition Using Kernel Methods, Advances in Neural Information Processing Systems, T. Diederich, S. Becker, Z. Ghahramani, Eds., 2002, vol. 14, 8 pages T.F. Cootes, K. Walker, C.J. Taylor, View-Based Active Appearance Models, Proc. of the IEEE International Conference on Automatic Face and Gesture Recognition, March 2000, Grenoble, France, pp T.F. Cootes, K. Walker, C.J. Taylor, View-Based Active Appearance Models, Proc. of the IEEE International Conference on Automatic Face and Gesture Recognition, March 2000, Grenoble, France, pp V. Blanz, T. Vetter, Face Recognition Based on Fitting a 3D Morphable Model, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 25, No. 9, September 2003, pp V. Blanz, T. Vetter, Face Recognition Based on Fitting a 3D Morphable Model, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 25, No. 9, September 2003, pp B. Moghaddam, J.H. Lee, H. Pfister, R. Machiraju, Model-Based 3D Face Capture with Shape-from-Silhouettes, Proc. of the IEEE International Workshop on Analysis and Modeling of Faces and Gestures, AMFG, 17 October 2003, Nice, France, pp B. Moghaddam, J.H. Lee, H. Pfister, R. Machiraju, Model-Based 3D Face Capture with Shape-from-Silhouettes, Proc. of the IEEE International Workshop on Analysis and Modeling of Faces and Gestures, AMFG, 17 October 2003, Nice, France, pp J. Lee, B. Moghaddam, H. Pfister, R. Machiraju, Finding Optimal Views for 3D Face Shape Modeling, Proc. of the International Conference on Automatic Face and Gesture Recognition, FGR2004, May 2004, Seoul, Korea, pp J. Lee, B. Moghaddam, H. Pfister, R. Machiraju, Finding Optimal Views for 3D Face Shape Modeling, Proc. of the International Conference on Automatic Face and Gesture Recognition, FGR2004, May 2004, Seoul, Korea, pp