1. Face Detection & Recognition
Dr. Gökhan Şengül

2. Topics Why face recognition?
What is difficult about real-time face recognition?
In general how is face recognition done?
Face detection…
Eigenfaces
Other face recognition algorithms
Future of face recognition

3. Application areas Security Personal information access
Fight terrorism
Find fugitives
Personal information access
ATM
Sporting events
Home access (no keys or passwords)
Any other application that would want personal identification
Improved human-machine interaction
Personalized advertising

4. System Requirements
Want the system to be inexpensive enough to use at many locations
Match within seconds
Before the person walks away from the advertisement
Before the fugitive has a chance to run away
Ability to handle a large database
Ability to do recognition in varying environments

5. Advantages of face recognition
Face images can be acquired from a long range
Many databases are available
Acceptable by people
People are willing to share their face images in public domain, e.g. facebook

6. What Is Difficult About Face Recognition?
Lighting variation
Orientation variation (face angle)
Size variation
Large database
Processor intensive
Time requirements

7. Intra-class variations

8. FERET DATABASE
Contains images of 1,196 individuals, with up to 5 different images captured for each individual
Often used to test face recognition algorithms
Information on obtaining the database can be found here:

9. Facial Features Level1 details
Consists of gross facial characteristics that are easily observable .i.e: general geometry of the face and global skin color
Can be used to discriminate elongated faces, faces exhibiting male and female characteristics, faces from different races

10. Facial Features Level2 details
Consists of localized face information such as the structure of the face components (e.g. eyes) , and the relationship between facial components, and the precise shape of the face
Essential for accurate face recognition
They require higher resolution face images (30 to 75 IPD)

11. Facial Features Level3 details
Consists of unstructured, micro level features on the face
Scars, freckles, skin distortion, and moles
Can be used for the discrimination of identical twins

12. Facial Features

13. General Face Recognition Steps
Face Detection
Face Normalization
Face Identification

14. General Face Recognition Process

15. Face detection and recognition
“Sally”

16. Applications of Face Recognition
Digital photography
Surveillance
Album organization

17. Applications of Face Recognition
Digital photography

18. Applications of Face Recognition
Digital photography
Surveillance

19. Sensors Used for image capture
Standard off-the-shelf PC cameras, webcams.
Requirements:
* Sufficient processor speed (main factor)
* Adequate Video card.
* 320 X 240 resolution.
* 3-5 frames per second.
( more frames per second and higher resolution lead to a better performance.)
One of the cheaper, inexpensive technologies starting at $ 50.

20. Sensors

21. Sensors
Face images captured in the visible and near-infrared spectra at different wavelengths.

22. Sensors

23. What is Face Detection?
Given an image, tell whether there is any human face, if there is, where is it(or where they are).

24. What is Face Detection?

25. Importance of Face Detection
The first step for any automatic face recognition system system
First step in many Human Computer Interaction systems
Expression Recognition
Cognitive State/Emotional State Recognition
First step in many surveillance systems
Tracking: Face is a highly non rigid object
A step towards Automatic Target Recognition(ATR) or generic object detection/recognition
Video coding……

26. Face Detection: current state
State-of-the-art:
Front-view face detection can be done at >15 frames per second on 320x240 black-and-white images on a 700MHz PC with ~95% accuracy.
Detection of faces is faster than detection of edges!
Side view face detection remains to be difficult.

27. Face Detection: challenges
Out-of-Plane Rotation: frontal, 45 degree, profile, upside down
Presence of beard, mustache, glasses etc
Facial Expressions
Occlusions by long hair, hand
In-Plane Rotation
Image conditions:
Size
Lighting condition
Distortion
Noise
Compression

28. Viola Jones Face Detector
It scans through the input image with detection windows of different sizes
Decides whether each window contains a face or not

29. Viola Jones Face Detector
The existence of a face candidate is decided by applying a classifier to simple local features derived using rectangular filters
The feature values are obtained by computing the difference between the sum of the pixel intensities in the light and dark rectangular regions

30. Viola Jones Face Detector

31. Viola Jones Face Detector

32. Viola Jones Face Detector

33. Feature Extraction and Matching
Knowledge-based methods:
Encode what constitutes a typical face, e.g., the relationship between facial features
Texture-based methods
The models are learned from a set of training images that capture the representative variability of faces.
Model based methods
attempt to build 2D or 3D face models that facilitate matching of face images in the presence of pose variations

34. Feature Extraction and Matching
Appearance-based methods
The generate a compact representation of the entire face region in the acquired image by mapping the high-dimensional face image into a lower dimensional sub-space. This sub-space is defined by a set of representative basis vectors, which are learned using a training set of images…
PCA (Principal Component Analysis)
LDA (Linear Discriminant Analysis)
ICA (Independent Component Analysis)

35. Feature Extraction and Matching
Model based methods
attempt to build 2D or 3D face models that facilitate matching of face images in the presence of pose variations
2D Face models
Face Bunch Graphs (FBG)
Active Appearance Model (AAM)
3D Face models

36. Feature Extraction and Matching
Texture-based methods:
try to find robust local features that are invariant to pose or lighting variations.
Examples of such features include
gradient orientations
Local Binary Patterns (LBP).

37. Feature Extraction and Matching

38. Appearance-based face recognition
are based on the idea of representing the given face image as a function of different face images available in the training set, or
as a function of a few basis faces.
the pixel value at location (x,y) in a face image can be expressed as a weighted sum of pixel values in all the training images at (x,y).
the goal in linear subspace analysis is to find a small set of most representative basis faces.
Any new face image can be represented as a weighted sum of the basis faces and two face images can be matched by directly comparing their vector of weights.

39. Principal Component Analysis (PCA)

40. Principal Component Analysis (PCA)

41. Principal Component Analysis (PCA)

42. Linear Discriminant Analysis (LDA)
Explicitly uses the class label of the training data and conducts subspace analysis with the objective of minimizing intra-class variations and maximizing inter-class variations
can generally be expected to provide more accurate face recognition when sufficient face image samples for each user are available during training

43. Linear Discriminant Analysis (LDA)

44. Linear Discriminant Analysis (LDA)

45. Linear Discriminant Analysis (LDA)

46. Feature Extraction and Matching

47. Model based face recognition
It tries to derive a pose-independent representation of the face images that can enable matching of face image across different poses.
These schemes typically require the detection of several fiducial or landmark points in the face (e.g., corners of eyes, tip of the nose, corners of the mouth, homogeneous regions of the face, and the chin)

48. Elastic Bunch Graph Matching
represents a face as a labeled image graph with each node being a fiducial or landmark point on the face.
While each node of the graph is labeled with a set of Gabor coefficients (also called a jet) that characterizes the local texture information around the landmark point, the edge connecting any two nodes of the graph is labeled based on the distance between the corresponding fiducial points.

49. Elastic Bunch Graph Matching
First stage: the designer has to manually mark the desired fiducial points and define the geometric structure of the image graph for one (or a few) initial image(s). The image graphs for the remaining images in the training set can be obtained semi-automatically

50. Elastic Bunch Graph Matching
second stage: a FBG is obtained from the individual image graphs by combining a representative set of individual graphs in a stack-like structure.
A set of jets corresponding to the same fiducial point is called a bunch. For example, an eye bunch may include jets from open, closed, male and female eyes, etc.
An edge between two nodes of the FBG is labeled based on the average distance between the corresponding nodes in the training set.

51. Elastic Bunch Graph Matching
Given a FBG, the fiducial points for a new face image are found by maximizing the similarity between a graph fitted to the given image and the FBG of identical pose. This process is known as Elastic Bunch Graph Matching (EBGM) and it consists of the following three steps:

52. Elastic Bunch Graph Matching

53. Elastic Bunch Graph Matching

54. Feature Extraction and Matching

55. Texture-Based Face Recognition
Based on the analysis of local textures
Two different approach:
SIFT (Scale-Invariant Feature Transform)
LBP (Local Binary Pattern)

56. Scale Invariant Feature Transform
Computation of SIFT features consists of two stages: (a) key point extraction, and (b) descriptor calculation in a local neighborhood at each key point.
The descriptor is usually a histogram of gradient orientations within a local neighborhood

57. Scale Invariant Feature Transform

58. Local Binary Pattern (LBP)
LBP features are usually obtained from image pixels of a 3×3 neighborhood region
MLBP
Multiscale LBP

59. Local Binary Pattern (LBP)

60. Local Binary Pattern (LBP)

61. Local Binary Pattern (LBP)
After LBP encoding of each pixel, the face image is divided into several smaller windows and the histogram of local binary patterns in each window is computed.
The number of bins in the histogram is 8 and 2P for the basic LBP and MLBP, respectively.
A global feature vector is then generated by concatenating histograms of all the individual windows and normalizing the final vector.
Finally, two face images can be matched by computing the similarity (or distance) between their feature vectors.

62. Face Databases