1. Recognition of Faces and Facial Attributes using Accumulative Local Sparse Representations
Domingo Mery
Department of Computer Science
Universidad Católica de Chile
Sandipan Banerjee
Department of Computer Science & Engineering
University of Notre Dame

2. Agenda
Motivation
Proposed method
Experiments
Conclusions

3. Agenda
Motivation
Proposed method
Experiments
Conclusions

4. class 1 class 2 class 3 : : : LEARNING TESTING class description
classifier’s design
LEARNING
TESTING
description
classification
class
query image

5. class 3 class 2 class 1 : LEARNING TESTING class description
classification
class
classifier’s design
query image

6. Are all parts of the face important?
Important for gender.
This problem gives rise to some interesting questions
This problem raises some interesting questions
Some interesting questions arise out of this problem
Important for race.
Important for expression.

7. Are all parts of the face important?
This problem gives rise to some interesting questions
This problem raises some interesting questions
Some interesting questions arise out of this problem
Important for Mary.

8. Are all parts of the face important?
This problem gives rise to some interesting questions
This problem raises some interesting questions
Some interesting questions arise out of this problem
Important for Mary and Miguel.
Not important at all!!!
Important for Miguel.

9. Agenda
Motivation
Proposed method
Experiments
Conclusions

10. SRC Sparse Representation Classification
Our method is based on...
SRC Sparse Representation Classification

11. Gallery
Subject Subject Subject Subject Subject-k
. . .

12. Gallery = Dictionary
Subject Subject Subject Subject Subject-k
. . .

13. . . . Gallery = Dictionary Sparse Representation Query
Subject Subject Subject Subject Subject-k
. . .
Sparse
Representation
Query

14. . . . Gallery = Dictionary 0.6 0.3 0.1 Sparse Representation Query
Subject Subject Subject Subject Subject-k
. . .
0.6
0.3
0.1
Sparse
Representation
Query

15. . . . Gallery = Dictionary 0.6 0.3 0.1 Sparse Representation Query
Subject Subject Subject Subject Subject-k
. . .
0.6
0.3
0.1
Sparse
Representation
Query

16. + + . . . Gallery 0.6 0.3 0.1 Sparse Representation 0.3 0.6 0.1 Query
Subject Subject Subject Subject Subject-k
. . .
0.6
0.3
0.1
Sparse
Representation
Query

17. Gallery
Subject Subject Subject Subject Subject-k
. . .
The query image is represented as a linear combination of few images of the gallery.
Query

18.
Query

19.
Query -
Not similar: reconstruction error is high

20.
Query -
Very similar: reconstruction error is very low

21. Query
Query is classified as this subject
In SRC, the query is classified as the subject with the lowest reconstruction error

22. ALSR Accumulative Local Sparse Representation [ PROPOSED METHOD ]

23. Our approach uses Patches!
description
selection
classification ID
query image

24. . . . . . . class i class 1 class k : : : LEARNING TESTING class : : :
description
description
description
classifier’s design
LEARNING
TESTING
description
classification
class
query image

25. . . . . . . class i class 1 class k : : : LEARNING TESTING : : :
description
description
description
dictionary 1
dictionary i
dictionary k
LEARNING
TESTING
Sparse representations have been widely used in many computer vision problems such as face recognition.
We build a dictionary from the gallery images.
We reconstruct the query image using a sparse combination of the dictionary.
We recognize the class by searching the minimal reconstruction error.
description
sparse
representation
classification
class
query image

26. . . . . . . class i class 1 class k : : : LEARNING TESTING class : : :
description
description
description
dictionary 1
dictionary i
dictionary k
LEARNING
TESTING
description
SRC
class
query image

27. . . . . . . class 1 class i class k : : : : : : LEARNING TESTING class : : : : : :
description
description
description
dictionary 1
dictionary i
dictionary k
LEARNING
TESTING
description
SRC
class
query image

28. . . . . . . class 1 class i class k : : : : : : LEARNING TESTING class : : : : : :
description
description
description
dictionary 1
dictionary i
dictionary k
LEARNING
TESTING
description
SRC
class
query image

29. . . . . . . for each test patch : for all test patches class 1 class i
class k : : : : : :
description
description
description
dictionary 1
dictionary i
dictionary k
LEARNING
TESTING
Huge dictionary
Sparse representation could be very time consuming
for each test patch
for all test patches
majority vote :
class
description
SRC
query image

30. . . . . . . for each test patch : for all test patches class 1 class i
class k : : : : : :
description
description
description
dictionary 1
dictionary i
dictionary k
LEARNING
TESTING
for each test patch
for all test patches
majority vote :
class
description
selection
of best
dictionaries
SRC
query image

31. . . . . . . for each test patch : for all test patches class 1 class i
class k : : : : : :
description
description
description
dictionary 1
dictionary i
dictionary k
LEARNING
TESTING
for each test patch
for all test patches
majority vote :
class
description
selection
of best
dictionaries
SRC
score
query image

32. Visual Vocabulary & Stop List
description :
class k
class i
class 1
Visual Vocabulary & Stop List
dictionary 1
dictionary i
dictionary k
LEARNING
TESTING
for each test patch
for all test patches
majority vote :
class
description
selection
of best
dictionaries
SRC
face mask
score
query image

33. . . . . . . Gallery For each image of the gallery: Dictionary
Subject Subject Subject Subject Subject-k
. . .
For each image of the gallery:
Dictionary
. . .
Patches of
Subject-1
Patches of
Subject-2
Patches of
Subject-k
+ position (x,y) of each patch

34. . . . Gallery Patches of Class-1 Class-2 Class-k General dictionary D
Class Class Class Class Class-k
Gallery
Patches of
Class-1
Class-2
Class-k
General dictionary D

35. Dictionary
Query

36. i For patch i 0. Original dictionary
1. Selection of the nearest patches
(using (x,y) information)
2. Selection of the most similar patches
(using intensity information)
3. Sparse Representation
0.5
0.3
0.2
Contribution si 4.
0.8
...
0.2 i
For patch i

37. i D Dn Ds For patch i 0. Original dictionary
1. Selection of the nearest patches
(using (x,y) information) Dn
2. Selection of the most similar patches
(using intensity information) Ds
3. Sparse Representation of yi using Ds
0.5
0.3
0.2 xi
Contribution i 4. si
0.8
...
0.2 i
Patch yi
Neighborhood of yi
For patch i

38. Subject
Contribution of each patch: 1 2 3 N
0.1
0.5
0.2
0.3
0.4
0.6 :
0.7 z
1.1
14.1
0.7
1.5
TOTAL
Query
classified as #2

39. Query Patches Contribution p 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 0.2
0.1
0.3
0.4
0.6
0.7
0.5 1 12
Query

40. x x x x Patches that are not discriminative can be removed
Patches Mask Contribution p 1 2 3 4 5 6 7 8 9 10 11 12 q 1 1 2 3 4
0.2
0.1
0.3
0.4
0.6
0.7
0.5
Patches that are not discriminative can be removed x x
Face masks used in
our experiments x x
Query

41. Patches that are not discriminative can be removed
Patches Mask Contribution p 1 2 3 4 5 6 7 8 9 10 11 12 q 1 1 2 3 4 -
0.1
0.3
0.2
0.4
0.6
0.5
Patches that are not discriminative can be removed
Query

42. SCI: Sparsity Concentration Index (score)
Patches Mask Contribution SCI p 1 2 3 4 5 6 7 8 9 10 11 12 q 1 1 2 3 4 -
0.1
0.3
0.2
0.4
0.6
0.5
SCI -
0.2
0.3
0.4
0.8
0.5
SCI: Sparsity Concentration Index (score)
Query

43. SCI: Sparsity Concentration Index > 0.25
Patches Mask Contribution SCI p 1 2 3 4 5 6 7 8 9 10 11 12 q 1 1 2 3 4 -
0.1
0.3
0.2
0.4
0.6
0.5
SCI -
0.2
0.3
0.4
0.8
0.5
SCI: Sparsity Concentration Index > 0.25
Query

44. SCI: Sparsity Concentration Index > 0.25
Patches Mask Contribution SCI p 1 2 3 4 5 6 7 8 9 10 11 12 q 1 1 2 3 4 -
0.1
0.4
0.6
0.2
0.3
0.5
SCI -
0.2
0.3
0.4
0.8
0.5
SCI: Sparsity Concentration Index > 0.25
Query

45. SCI: Sparsity Concentration Index > 0.25
Patches Mask Contribution SCI p 1 2 3 4 5 6 7 8 9 10 11 12 q 1 1 2 3 4 -
0.1
0.4
0.6
0.2
0.3
0.5
SCI -
0.3
0.4
0.8
0.5
SCI: Sparsity Concentration Index > 0.25
Query

46. Maximal value of each contribution
Patches Mask Contribution SCI max p 1 2 3 4 5 6 7 8 9 10 11 12 q 1 1 2 3 4 -
0.1
0.4
0.6
0.2
0.3
0.5
SCI -
0.3
0.4
0.8
0.5
max -
0.4
0.6
0.3
0.5
Maximal value of each contribution
Query

47. Each contribution is divided by its maximum
Patches Mask Contribution SCI max Normalization p 1 2 3 4 5 6 7 8 9 10 11 12 q 1 1 2 3 4 -
0.1
0.4
0.6
0.2
0.3
0.5
SCI -
0.3
0.4
0.8
0.5
max -
0.4
0.6
0.3
0.5 1 2 3 4 -
0.25
1.0
0.17
0.33
0.2
0.4
Each contribution is divided by its maximum
Query

48. The normalized contributions must be greater than 0.2
Patches Mask Contribution SCI max Normalization p 1 2 3 4 5 6 7 8 9 10 11 12 q 1 1 2 3 4 -
0.1
0.4
0.6
0.2
0.3
0.5
SCI -
0.3
0.4
0.8
0.5
max -
0.4
0.6
0.3
0.5 1 2 3 4 -
0.25
1.0
0.17
0.33
0.2
0.4
The normalized contributions must be greater than 0.2
Query

49. The normalized contributions must be greater than 0.2
Patches Mask Contribution SCI max Normalization p 1 2 3 4 5 6 7 8 9 10 11 12 q 1 1 2 3 4 -
0.1
0.4
0.6
0.2
0.3
0.5
SCI -
0.3
0.4
0.8
0.5
max -
0.4
0.6
0.3
0.5 1 2 3 4 -
0.25
1.0
0.33
0.4
The normalized contributions must be greater than 0.2
Query

50. The query is classified according the maximal normalized contribution
Patches Mask Contribution SCI max Normalization p 1 2 3 4 5 6 7 8 9 10 11 12 q 1 1 2 3 4 -
0.1
0.4
0.6
0.2
0.3
0.5
SCI -
0.3
0.4
0.8
0.5
max -
0.4
0.6
0.3
0.5 1 2 3 4 -
0.25
1.0
0.33
0.4
0.25
3.33
1.4
0.33
max
The query is classified according the maximal normalized contribution
Query

51. The code of the MATLAB implementation is available on our webpage:
> Material > ALSR

52. Agenda
Motivation
Proposed method
Experiments
Conclusions

53. Experiments Face Recognition in LFW Gender Recognition in AR
Expression Recognition in Oulu-CASIA

54. Face Recognition in LFW [ PROTOCOL ]
The gallery has 143 subjects with at least 11 images per subject (10 for training, the rest for testing). There are 1430 images for training and 2744 for testing.

55. Example in LFW Contributions per class Maximum for #117 1 ... 117 143
Images of the same subject in the gallery (subject #117).
Query image
Contributions per class
Maximum for #117

56. Example in LFW
Images of the same subject in the gallery (subject #98).

57. 80.8
In this table, we do not report deep learning methods that require millions of training images (for the sake of truth, VGG-Face in this experiment achieves 97.7%)

58. Gender Recognition in AR [ PROTOCOL ]
100 subjects (50 women and 50 men). For gender recognition, 14 non-occluded images per subject. In this experiment, the first 25 males and 25 females were used for training and the last 25 males and 25 females were used for testing.

60. Expression Recognition in Oulu-CASIA [ PROTOCOL ]
Six different facial expressions (surprise, happiness, sadness, anger, fear and dis- gust) under normal illumination from 80 subjects (59 males and 21 females) ranging from 23 to 58 years in age. The dataset contains 480 sequences: the first 9 images of each sequence are not considered, the first 40 individuals are taken as training subset and the rest as testing.

62. Agenda
Motivation
Proposed method
Experiments
Conclusions

63. We presented a new algorithm that is able to recognize faces and facial attributes automatically from face images captured under less constrained conditions including some variability in ambient lighting, pose, expression, size of the face and distance from the camera.
The robustness of our algorithm is due to two reasons:
The dictionary used in the recognition corresponds to a rich collection of representations of relevant parts which were selected using closeness and similarity criteria.
The testing stage is based on accumulative sparse contributions according to location and relevance criteria.
We believe that this new approach can be used to solve other kind of computer vision problems in which there are similar unconstrained conditions and a huge number of training images is not available.
In the future, we will train our own deep learning network to obtain a better description of the patches, and we will learn the face image masks from training data, instead of manual selection.

64. Recognition of Faces and Facial Attributes using Accumulative Local Sparse Representations
Domingo Mery
Department of Computer Science
Universidad Católica de Chile
Sandipan Banerjee
Department of Computer Science & Engineering
University of Notre Dame