1. Outline
Peter N. Belhumeur, Joao P. Hespanha, and David J. Kriegman, “Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp , 1997.

2. The Goal
Face recognition that is insensitive to large variations in lighting and facial expressions
Note that lighting variability here includes lighting intensity, direction, and number of light sources
November 15, 2018
Computer Vision

3. The Difficulty
It is difficult because the same person with the same facial expression, and seen from the same viewpoint, can appear dramatically different when light sources illuminate the face from different directions
November 15, 2018
Computer Vision

4. Observation
All of the images of a Lambertian surface, taken from a fixed viewpoint, but under varying illumination, lie in a 3D linear subspace of the high-dimensional image space
Image formulation can be modeled by
For a Lambertian surface, the amount of reflected light does not depend on the viewing direction, but only on the cosine angle between the incidence light ray and the normal of the surface
So, for Lambertian surfaces, we have
November 15, 2018
Computer Vision

5. Observation
Therefore, in the absence of shadowing, given three images of a Lambertian surface from the same viewpoint taken under three known, linearly independent light source directions, the albedo and surface normal can be recovered
One can reconstruct the image of the surface under an arbitrary lighting direction by a linear combination of three original images
November 15, 2018
Computer Vision

6. 3D Linear Space Example
November 15, 2018
Computer Vision

7. 3D Linear Space Example
November 15, 2018
Computer Vision

8. The Problem Statement
Given a set of face images labeled with the person’s identity and un unlabeled set of face images from the same group of people, identify each person in the test images
November 15, 2018
Computer Vision

9. Correlation Nearest neighbor in the image space
If all the images are normalized to have zero mean and unit variance, it is equivalent of choosing the image in the learning set that best correlates with the test image
Due to the normalization process, the result is independent of light source intensity
November 15, 2018
Computer Vision

10. Correlation
Covariance
Correlation
November 15, 2018
Computer Vision

11. Correlation Problems with correlation
If the images are gathered under varying light conditions, then the corresponding points in the image space may not be tightly clustered
Computationally, it is expensive to compute correlation between two images
All the images have to be stored, which can require a large amount of storage
November 15, 2018
Computer Vision

12. Eigenfaces
As we discussed last time, we can reduce the computation by dimension reduction using PCA
Suppose we have a set of N images and there are c classes
We define a linear transformation
The total scatter of the training set is given by
November 15, 2018
Computer Vision

13. Eigenfaces
For PCA, it chooses to maximize the total scatter of the transformed feature vectors , which is
Mathematically, we have
November 15, 2018
Computer Vision

14. Eigenfaces
When lighting changes, the total scatter is due to the between-class scatter that is useful for classification and also due to the within-class scatter, which is unwanted for classification purposes
When lighting changes, much of the variation from one image to the next is due to the illumination changes
An ad-hoc way of dealing with this problem is to discard the three most significant principal components, which reduces the variations due to lighting
November 15, 2018
Computer Vision

15. Linear Subspaces
Note that all images of a Lambertian surface under different lighting lie in a 3D linear subspace
For each face, use three or more images taken under different lighting directions to construct a 3D basis for the linear subspace
To perform recognition, we simply compute the distance to a new image to each linear subspace and choose the face corresponding to the shortest distance
If there is no noise or shadowing, it would achieve error free classification under any lighting conditions
November 15, 2018
Computer Vision

16. Fisherfaces
Using Fisher’s linear discriminant to find class-specific linear projections
More formally, we define the between-class scatter
The within-class scatter
Then we choose to maximize the ratio of the determinant of the between-class scatter matrix to the within-class scatter of the projected samples
November 15, 2018
Computer Vision

17. Fisherfaces
That is,
November 15, 2018
Computer Vision

18. Comparison of PCA and FDA
November 15, 2018
Computer Vision

19. Fisherfaces Singularity problem
The within-class scatter is always singular for face recognition
This problem is overcome by applying PCA first, which can be called PCA/LDA
November 15, 2018
Computer Vision

20. Experimental Results Variation in lighting November 15, 2018
Computer Vision

21. Experimental Results
November 15, 2018
Computer Vision

22. Experimental Results
November 15, 2018
Computer Vision

23. Experimental Results
November 15, 2018
Computer Vision

24. Variations in Facial Expression, Eye Wear, and Lighting
November 15, 2018
Computer Vision

25. Variations in Facial Expression, Eye Wear, and Lighting
November 15, 2018
Computer Vision

26. Variations in Facial Expression, Eye Wear, and Lighting
November 15, 2018
Computer Vision

27. Variations in Facial Expression, Eye Wear, and Lighting
November 15, 2018
Computer Vision

28. Variations in Facial Expression, Eye Wear, and Lighting
November 15, 2018
Computer Vision

29. Variations in Facial Expression, Eye Wear, and Lighting
November 15, 2018
Computer Vision

30. Glasses Recognition Glasses / no glasses recognition November 15, 2018
Computer Vision