1. Face Recognition: An Introduction

2. Face Recognition Face is the most common biometric used by humans
Applications range from static, mug-shot verification to a dynamic, uncontrolled face identification in a cluttered background
Challenges:
automatically locate the face
recognize the face from a general view point under different illumination conditions, facial expressions, and aging effects O
O Face recognition can be categorized into appearance-based, geometry-based, and hybrid approaches.

3. Authentication vs Identification
Face Authentication/Verification (1:1 matching)
Face Identification/Recognition (1:N matching)

4. Applications
 Access Control

5. Applications Face Scan at Airports
 Video Surveillance (On-line or off-line)
Face Scan at Airports

6. Face Recognition Application

7. Why is Face Recognition Hard?
Many Faces of the same person

8. Face Recognition Difficulties
Identify similar faces (inter-class similarity)
Accommodate intra-class variability due to:
head pose
illumination conditions
expressions
facial accessories
aging effects O
O Face recognition can be categorized into appearance-based, geometry-based, and hybrid approaches.
Cartoon faces

9. Inter-class Similarity
Different persons may have very similar appearance
news.bbc.co.uk/hi/english/in_depth/americas/2000/us_elections O
O Face recognition can be categorized into appearance-based, geometry-based, and hybrid approaches.
Twins
Father and son

10. Intra-class Variability
Faces with intra-subject variations in pose, illumination, expression, accessories, color, occlusions, and brightness O
O Face recognition can be categorized into appearance-based, geometry-based, and hybrid approaches.

11. Sketch of a Pattern Recognition Architecture
Feature
Extraction
Classification
Image
(window)
Object
Identity
Feature
Vector

12. Example: Face Detection
Scan window over image
Classify window as either:
Face
Non-face
Classifier
Window
Face
Non-face

13. Example: Finding skin
Skin has a very small range of (intensity independent) colors, and little texture
Compute an intensity-independent color measure, check if color is in this range, check if there is little texture (median filter)
See this as a classifier - we can set up the tests by hand, or learn them.
get class conditional densities (histograms), priors from data (counting)
Classifier is

15. Face Detection

16. Face Detection Algorithm
Lighting Compensation
Color Space Transformation
Skin Color Detection
Input Image
Variance-based Segmentation
Connected Component &
Grouping
Face Localization
Eye/ Mouth Detection O
O Face recognition can be categorized into appearance-based, geometry-based, and hybrid approaches.
Face Boundary Detection
Verifying/ Weighting
Eyes-Mouth Triangles
Facial Feature Detection
Output Image

17. Face Recognition: 2-D and 3-D
Time
(video)
2-D
Face
Database
2-D
Recognition
Data
Recognition
Comparison
3-D
3-D
Play this, it’s animated – Green denotes topics covered subsequent slides
Prior knowledge
of face class

18. Taxonomy of Face Recognition
Algorithms
Pose-dependency
Pose-dependent
Pose-invariant
Viewer-centered Images
Object-centered Models
Face representation
Matching features
Appearance-based (Holistic)
Hybrid
Feature-based (Analytic)

19. Image as a Feature Vectorx 1 2 3
Consider an n-pixel image to be a point in an n-dimensional space, x Rn.
Each pixel value is a coordinate of x.

20. Nearest Neighbor Classifier
{ Rj } are set of training images. x 1 2 3 R1 R2 I

21. Comments Sometimes called “Template Matching”
Variations on distance function (e.g. L1, robust distances)
Multiple templates per class- perhaps many training images per class.
Expensive to compute k distances, especially when each image is big (N dimensional).
May not generalize well to unseen examples of class.
Some solutions:
Bayesian classification
Dimensionality reduction

22. Eigenfaces
A set of eigenfaces can be generated by performing a mathematical process called principal component analysis (PCA) on a large set of images depicting different human faces.
Informally, eigenfaces can be considered a set of "standardized face ingredients", derived from statistical analysis )(covariance matrix ) of many pictures of faces. Any human face can be considered to be a combination of these standard faces.

23. Eigenfaces
For example, one's face might be composed of the average face plus 10% from eigenface 1, 55% from eigenface 2, and even -3% from eigenface 3.
The eigenfaces that are created will appear as light and dark areas that are arranged in a specific pattern.

24. Eigenfaces
These eigenfaces can now be used to represent both existing and new faces: we can project a new (mean-subtracted) image on the eigenfaces and thereby record how that new face differs from the mean face.
The eigenvalues associated with each eigenface represent how much the images in the training set vary from the mean image in that direction.

25. Eigenfaces
We lose information by projecting the image on a subset of the eigenvectors, but we minimize this loss by keeping those eigenfaces with the largest eigenvalues.
For instance, if we are working with a 100 x 100 image, then we will obtain 10,000 eigenvectors. In practical applications, most faces can typically be identified using a projection on between 100 and 150 eigenfaces, so that most of the 10,000 eigenvectors can be discarded.

26. How do you construct Eigenspace?
[ ] [ ]
[ x1 x2 x3 x4 x5 ] W
Construct data matrix by stacking vectorized images and then apply Singular Value Decomposition (SVD)

27. Eigenfaces

28. Eigenfaces Modeling Recognition
Given a collection of n labeled training images,
Compute mean image and covariance matrix.
Compute k Eigenvectors (note that these are images) of covariance matrix corresponding to k largest Eigenvalues.
Project the training images to the k-dimensional Eigenspace.
Recognition
Given a test image, project to Eigenspace.
Perform classification to the projected training images.

29. Eigenfaces: Training Images

30. Eigenfaces
Mean Image
Basis Images

31. Difficulties with PCA Projection may suppress important detail
smallest variance directions may not be unimportant
Method does not take discriminative task into account
typically, we wish to compute features that allow good discrimination
not the same as largest variance

32. Fisherfaces: Class specific linear projection
An n-pixel image xRn can be projected to a low-dimensional feature space yRm by
y = Wx
where W is an n by m matrix.
Recognition is performed using nearest neighbor in Rm.
How do we choose a good W?

33. PCA & Fisher’s Linear Discriminant
Between-class scatter
Within-class scatter
Total scatter
Where
c is the number of classes
i is the mean of class i
| i | is number of samples of i.. 1 2  1 2
Use this slide to explain the three types of scatter

34. PCA & Fisher’s Linear Discriminant
PCA (Eigenfaces)
Maximizes projected total scatter
Fisher’s Linear Discriminant
Maximizes ratio of projected between-class to projected within-class scatter 2 1
Point to emphasize is that
PCA maximizes the projected total scatter – I.e., it presserves the information in the training set, and is optimal in a least squares sense for reconstruction. But in dropping dimensions, it may smear classes together.
FLD trades off two desirable effects for recognition.
A. The within class scatter over all classes is minimized – this makes classification using nearest neighbor effective.
B. The between class scatter is maximized – this causes classes to be far apart in the feature space.
FLD

35. Four Fisherfaces Database

36. Eigenfaces and Fisherfaces