1. KINSHIP CLASSIFICATION BY MODELING FACIAL FEATURE HEREDITY
IEEE International Conference on Image Processing 2013
KINSHIP CLASSIFICATION BY MODELING FACIAL FEATURE HEREDITY
Definition of the new challenge
Kinship classification: recognizing the family that a query person belongs to from a set of families.
Ruogu Fang1, Andrew C. Gallagher1
Tsuhan Chen1, Alexander Loui2
1 Cornell University
2 Eastman Kodak Company

2. KINSHIP CLASSIFICATION BY MODELING FACIAL FEATURE HEREDITY
Problem Definition:
Recognize the family that a query person belongs to from a set of families.
Solution:
Reconstruct the query face from a mixture of parts from a set of family members for the recognition.
Motivation:
Genetic model of reproduction using the mathematical tool of sparsity.
Every face image in in collage represents a family.

3. The facial feature heredity also follow the model of genetics.
A genetic perspective
DNA
Why do we look like the way that we do?
DNA
How are our appearances affected by ancestors?
Inheritance and mutation
Facial features are part of the appearance.
The story:
-computer vision has used many representations for faces.
- but there is one ”natural” representation, DNA,
-DNA affects every aspect of how our bodies appear (along with environment)
-In fact, you can go from DNA to appearance...
-Therefore, we would like to model the process of genetics in our kinship work. We do this with sparsity.
Facial features are part of the appearance
The facial feature heredity also follow the model of genetics.

4. Mendel’s Laws I
Law of Random Segregation: For every particular trait, one randomly selected allele from each parent is passed down to the offspring.
B: Brown eyes (dominant)
b: blue eyes (recessive)
b b
Each facial feature of an individual can be represented by a sparse combination of the relatives with this feature.
B b Bb Bb BB Bb B b B b
Sparse selection:
For each gene, a child receives a pair of alleles, one allele from each parent. The visible appearance (phenotype) of the individual depends on which alleles are dominant and which are recessive. This process is a sparse process: for various genes, the displayed phenotype for each gene is a sparse selection from among the parents.
Explain what are dominant and recessive genes
Every trait comes from a relative in the family bb bb Bb bb

5. ? family sparsity Few families are selected.
Few people from few families are selected. ?

6. Energy function
For one family, given sufficient training samples of a family (m = feature length, n = number of training samples)
A new sample from the same family
Approximately lies in the linear span of the family member samples associated with this family
For all unrelated families,
L2,1 norm: Family sparsity term
N= # families
L21 norm: few families involved in the reconstruction
L1 norm: selecting only a few images with similar illumination, pose and expression.
Few people in few families are selected.
Latex:
D_j=[d_{j,1},d_{j,2},\ldots,d_{j,n}]\in R^{m\times n}
y\approx D_j\alpha
L1 norm: Individual sparsity term
(illumination, pose and expression)

7. The facial features should be analyzed independently.
Mendel’s Laws II
Law of Independent Assortment: Genes of separate traits are passed down independently from parents to offspring.
The facial features should be analyzed independently.
Credit: Northeast Medical School

8. Independence of facial parts
For each part, a part-based dictionary is built.
We segment the facial image into 12 parts. The location and the size of the 12 facial parts are determined from the annotation of one researcher on a template face and applied to all the other facial images which have been aligned to the same template.

9. classification
Reconstruction error for part p from family j …
# families
Error
Remove outliers due to recessive genes
Choose three representative parts with smallest possible residues R.
Rank the normalized residues for all families on these three parts.
Sum the ranks and use the highest rank.
Error
Byproduct: Find the three most distinguishable features …
# families …
It is a heuristic method for the classification.
Latex
R_j^{(p)}=\|y^{(p)}-D_{G_j}^{(p)}\alpha_j^{(p)}\|_2^2
Error …
# families

10. Potential applications
Family Image Retrieval
Family Photo Album Distillation
Tag Your Family Members
From Sara Lee’s family?
From Kelly Ng’s family?
Social Websites: Auto Family Tagging
Find Lost Relatives

11. Family 101 database 607 Individuals 14,816 Images
101 Different Families
607 Individuals
14,816 Images
Download:
Kennedy
27 (410)
political, royal, wealthy and celebrity families
# people
# images

12. family database Collection
Kennedy Family
27 Individuals
Caroline Kennedy
48 Images of Caroline Kennedy

13. Related databases Facts about Family101 Database Multiple generations
# of Family
# of People
# of Images
Highlights
References
Labeled Faces in the Wild (LFW)
5749
13,233
Unconstrained, “natural” variability in pose, lighting, etc.
[Hung et al. 2007]
PubFig
200
58,797
Real world, deep and large, celebrities and politicians
[Kumar et al. 2009]
Cornell Kinship Verification
150
300
Controlled parent-child pairs
[Fang et al. 2010]
UB KinFace 90
180
270
Child with young parent and old parent faces
[Xia et al. 2011]
Family101
101
(206 Nuclear)
607
14,816
Real world, family structure of 2-3 genenrations, variations of age, pose, illumination, expression, ethnicity, etc. Political, royal, wealthy and celebrity families.
This Work
Facts about Family101 Database
Multiple generations
Every nuclear family has 6 family members on average
Every individual has 24 images on average

14. Experiment setup Feature: Dense SIFT 16x16 Baseline
K nearest neighbors (KNN)
Support vector machine (SVM)
Sparse representation based recognition (SRC)
Unless specified in each scenario:
3 family members for training, 2 for testing.
20 families randomly selected for evaluation.
30 images/person for both training and testing.
Evaluation metric: Mean per-family accuracy

15. Exp 1: No. of families 3 family members for training
2 family members for testing
30 images/person for training/testing

16. Exp 2: No. of people for training
20 families randomly selected
30 images/person for training/testing

17. Facial Feature Matching
Task: Find the people with similar facial features to the query person.
Martin Sheen
High
Low
Hair
Eyes
Nose
Mouth
coefficients
Martin Sheen
Decide the family.
Pick up the top three images in the part-based dictionary D^(p) with the largest coefficients.
These images are regarded as the images of family relatives who have similar facial traits.
Test Images
Training Images

18. conclusion Motivation: biological process of inheritance
Mendel’s laws of random segregation and independent assortment
A new challenge: kinship classification
A new framework: reconstruct the query face from a mixture of parts from a set of families
A new dataset: Family 101

19. Future work Use family tree structure Hallucination
Hallucinate what the appearance of the father might be, just by looking at the differences between a child and her mother.
Hallucination: consider a child with her mother. Facial parts that are in common are explained by the mother’s passing of her alleles for those traits. For facial parts that are different, we expect that it is likely that those traits were inherited from the father.

20. Q & A KINSHIP CLASSIFICATION BY MODELING FACIAL FEATURE HEREDITY
Ruogu Fang
Andrew C. Gallagher
Q & A
Thank you!
Project Page and Dataset Download:
Tsuhan Chen
Alexander Loui

21. Face detection & alignment
Face Alignment: 6 Fiducial Points
Active Shape Model: 82 Facial Points
Face Detection