Outline Peter N. Belhumeur, Joao P. Hespanha, and David J. Kriegman, “Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 711-720, 1997.

The Goal Face recognition that is insensitive to large variations in lighting and facial expressions Note that lighting variability here includes lighting intensity, direction, and number of light sources November 15, 2018 Computer Vision

The Difficulty It is difficult because the same person with the same facial expression, and seen from the same viewpoint, can appear dramatically different when light sources illuminate the face from different directions November 15, 2018 Computer Vision

Observation All of the images of a Lambertian surface, taken from a fixed viewpoint, but under varying illumination, lie in a 3D linear subspace of the high-dimensional image space Image formulation can be modeled by For a Lambertian surface, the amount of reflected light does not depend on the viewing direction, but only on the cosine angle between the incidence light ray and the normal of the surface So, for Lambertian surfaces, we have November 15, 2018 Computer Vision

Observation Therefore, in the absence of shadowing, given three images of a Lambertian surface from the same viewpoint taken under three known, linearly independent light source directions, the albedo and surface normal can be recovered One can reconstruct the image of the surface under an arbitrary lighting direction by a linear combination of three original images November 15, 2018 Computer Vision

3D Linear Space Example November 15, 2018 Computer Vision

3D Linear Space Example November 15, 2018 Computer Vision

The Problem Statement Given a set of face images labeled with the person’s identity and un unlabeled set of face images from the same group of people, identify each person in the test images November 15, 2018 Computer Vision

Correlation Nearest neighbor in the image space If all the images are normalized to have zero mean and unit variance, it is equivalent of choosing the image in the learning set that best correlates with the test image Due to the normalization process, the result is independent of light source intensity November 15, 2018 Computer Vision

Correlation Covariance Correlation November 15, 2018 Computer Vision

Correlation Problems with correlation If the images are gathered under varying light conditions, then the corresponding points in the image space may not be tightly clustered Computationally, it is expensive to compute correlation between two images All the images have to be stored, which can require a large amount of storage November 15, 2018 Computer Vision

Eigenfaces As we discussed last time, we can reduce the computation by dimension reduction using PCA Suppose we have a set of N images and there are c classes We define a linear transformation The total scatter of the training set is given by November 15, 2018 Computer Vision

Eigenfaces For PCA, it chooses to maximize the total scatter of the transformed feature vectors , which is Mathematically, we have November 15, 2018 Computer Vision

Eigenfaces When lighting changes, the total scatter is due to the between-class scatter that is useful for classification and also due to the within-class scatter, which is unwanted for classification purposes When lighting changes, much of the variation from one image to the next is due to the illumination changes An ad-hoc way of dealing with this problem is to discard the three most significant principal components, which reduces the variations due to lighting November 15, 2018 Computer Vision

Linear Subspaces Note that all images of a Lambertian surface under different lighting lie in a 3D linear subspace For each face, use three or more images taken under different lighting directions to construct a 3D basis for the linear subspace To perform recognition, we simply compute the distance to a new image to each linear subspace and choose the face corresponding to the shortest distance If there is no noise or shadowing, it would achieve error free classification under any lighting conditions November 15, 2018 Computer Vision

Fisherfaces Using Fisher’s linear discriminant to find class-specific linear projections More formally, we define the between-class scatter The within-class scatter Then we choose to maximize the ratio of the determinant of the between-class scatter matrix to the within-class scatter of the projected samples November 15, 2018 Computer Vision

Fisherfaces That is, November 15, 2018 Computer Vision

Comparison of PCA and FDA November 15, 2018 Computer Vision

Fisherfaces Singularity problem The within-class scatter is always singular for face recognition This problem is overcome by applying PCA first, which can be called PCA/LDA November 15, 2018 Computer Vision

Experimental Results Variation in lighting November 15, 2018 Computer Vision

Experimental Results November 15, 2018 Computer Vision

Experimental Results November 15, 2018 Computer Vision

Experimental Results November 15, 2018 Computer Vision

Variations in Facial Expression, Eye Wear, and Lighting November 15, 2018 Computer Vision

Variations in Facial Expression, Eye Wear, and Lighting November 15, 2018 Computer Vision

Variations in Facial Expression, Eye Wear, and Lighting November 15, 2018 Computer Vision

Variations in Facial Expression, Eye Wear, and Lighting November 15, 2018 Computer Vision

Variations in Facial Expression, Eye Wear, and Lighting November 15, 2018 Computer Vision

Variations in Facial Expression, Eye Wear, and Lighting November 15, 2018 Computer Vision

Glasses Recognition Glasses / no glasses recognition November 15, 2018 Computer Vision