Preliminary Assessment of Discrimination of Twins in Photographs based on Facial Blemishes Nisha Srinivas 1, Matthew Pruitt 1, Gaurav Aggarwal 1, Patrick Flynn 1, Richard Vorder Bruegge 2 1 University of Notre Dame 2 Frederal Bureau of Investigation, Digital Evidence Lab

Problem Statement Investigate the usefulness of facial blemishes to distinguish between identical twins – Moles, Freckles, Scars, etc Determine – Whether facial blemishes and locations can be used to distinguish between identical twins – Whether the distributions of facial blemishes are “more similar” for identical twins than unrelated persons?

Facial Blemishes Types of facial blemishes – Mole – Freckle – Freckle Group – Pimple – Darkened Patch – Lightened Patch – Splotchiness – Birthmark – Raised Skin – Pockmark – Scars Linear Round Mole Freckle and Freckle Group Lightened Patch Darkened Patch Raised SkinScar (Round)PockmarkPimple

Proposed System Overview Manual Annotation Feature Extraction Geometric Normalization Point Cloud Matching Biometric Verification Performance Evaluation

Manual Annotation Display Module Annotation Module Tool Module

Facial Blemishes Identified by Observer 1

Facial Blemishes Identified by Observer 2

Facial Blemishes Identified by Observer 3

Facial Blemishes Identified by Observers Total Number of Facial Blemishes Annotated by each Observer Observer 1: 3785 Observer 2: 2311 Observer 3: 5100

Facial Blemishes Matching N NodesM Nodes Moles

Matching Contd. The Edges in the bipartite graph correspond to potential matches Each potential match has a cost associated with it which is a function of the euclidean distance between the centroids of the blemishes being compared.

Matching Contd. Match Similarity metric=Number of matches/Max(N,M)

Data Twin face images were collected at the Twins Days Festival in Twinsburg, Ohio in August High Resolution Images: 4310 rows x 2868 columns Dataset Attributes – Frontal (yaw=0), Indoor, No Glasses, Neutral Expression Number of Images: 295 – Number of Subjects: 152 – Number of Twins Pairs: 76 Terminology – Target set: “gallery” of persons to be recognized – Query set: a set of images of unidentified persons to be matched against the target set

Experimental Setup Perform two different experiments – Individual Observer Analysis Query set and Target set are annotated by same observer – Inter-Observer Analysis Query set is annotated by one observer and the Target set is annotated by another observer – Observer 1 vs Observer 2 – Observer 2 vs Observer 3 – Observer 3 vs Observer 1

Subset of Facial blemishes FM={moles, freckles, freckle group, pimple, birthmark, darkened patch, lightened patch, splotchiness, raised skin, pockmark, scar round, scar linear} FM1=FM-{pimple} FM2={moles, freckles} FM3={moles, freckles, pimple}

Twins vs Twins Setup: Query Set Target Set Subject 1, Twin A Subject 2, Twin B Subject 3, Twin A Subject 4, Twin B Match Comparison Non-Match Comparison

Individual Performance Evaluation- Observer 3

Match Comparison Non-Match Comparison Query Set Target Set Subject 1, Twin A Subject 2, Twin B Subject 3, Twin A Subject 4, Twin B Subject 5, Twin A All vs All Setup:

Individual Performance Evaluation- Observer 3

Comparison: All vs All and Twins vs Twins

Inter-Observer Performance Degradation in Performance when comparing facial marks annotated by different observers

Conclusion There appears a correlation between the distribution of facial blemishes across twins. The number of facial blemishes across twins appears to be similar. Facial blemishes can be used as a potential biometric signature. Consistent annotation is a challenging process – It is difficult to achieve consistency

Thank You This research was supported by – NIJ/OJP award 2009-DN-BX-K231 – FBI through TSWG/ARMY RDECOM contract W91CRB-08-C-0093