CVPR SLAM 2007 Using Group Prior to Identify People In Consumer Images Andrew C. Gallagher Tsuhan Chen Carnegie Mellon University Eastman Kodak Company June 18, 2007

CVPR SLAM 2007 The Problem Consumer image collections are growing exponentially each year. Consumers want to search for images based on whom the image contains. And they don’t like to label images! This is more than a face recognition problem. To best understand the semantics of who is in the images, we need to understand the people in the images.

CVPR SLAM 2007 Traditional Face Recognition Determines the assignment of each person independently Extract facial features Build a classifier that finds the most likely name, given the features. But this method does not take full advantage of the available information!

CVPR SLAM 2007 The Group Prior for Learning The Semantics of People in Images Determine the joint assignment of all people in the image to names, using the group prior. By the unique object constraint (UOC), an individual can appear only once in the image. The group prior characterizes the prior probability of certain groups of people appearing together in an image.

CVPR SLAM 2007 System Diagram Ambiguous Label Resolution Images (Faces) Ambiguous Labels Classifier Training Labeled Faces Recognize People Annotated Image Hannah Jonah Holly Andy Jonah Holly Jonah Jonah Holly Jonah Holly Jonah AndyHannah Unlabeled Image Group Prior Hannah Holly

CVPR SLAM 2007 Recognizing a Person When a single person is in the image: : the set of all unique names : a member of the set : the features from person image Posterior Probability Individual Prior Likelihood

CVPR SLAM 2007 Recognizing Multiple People The graph model represents the features and people in an image. The graph encodes the independence assumptions of our model. E.g. given the identity of a person, their features are independent of others in the image. p1p1 f1f1 p2p2 f2f2 pMpM fMfM … … The Group Prior

CVPR SLAM 2007 Recognizing Multiple People The joint probability function: p1p1 f1f1 p2p2 f2f2 pMpM fMfM … … The Group Prior : an index over the people in the image : the set of all features for all people : the set of people in the image : a subset of ; a particular assignment of a name to each person in. The Group PriorLikelihood

CVPR SLAM 2007 Estimating the Group Prior For pairs of names, the group prior is estimated by counting the number of images the pair appears, then normalizing. The group prior for 3 or more people is estimated according to the group prior pairwise graphical model. The Group Prior The Individual Prior

CVPR SLAM 2007 System Diagram Ambiguous Label Resolution Images (Faces) Ambiguous Labels Classifier Training Labeled Faces Recognize People Annotated Image Hannah Jonah Holly Andy Jonah Holly Jonah Jonah Holly Jonah Holly Jonah AndyHannah Unlabeled Image Group Prior Hannah Holly

CVPR SLAM 2007 Ambiguous Labels Ambiguous labels indicate who is in the image, but not which person is which name. A constrained clustering algorithm is used to ‘resolve’ the labels. The resolved labels are used to learn the feature distribution for each name. Andy Hannah Andy Hannah JonahHannahJonahHannah Jonah Holly Hannah JonahHolly JonahAndyJonah Andy Hannah Jonah Andy Hannah Jonah Holly Hannah Andy HannahHolly JonahAndy HannahJonah Holly HannahAndy HannahAndy Hannah Andy Hannah JonahHannahJonahHannah Jonah Holly Hannah JonahHolly JonahAndyJonah Andy Hannah Jonah Andy Hannah Jonah Holly Hannah Andy HannahHolly JonahAndy HannahJonah Holly HannahAndy JonahAndy Hannah Andy Hannah JonahHannahJonahHannah Jonah Holly Hannah JonahHolly JonahAndyJonah Andy Hannah Jonah Andy Hannah Jonah Holly Hannah Andy HannahHolly JonahAndy HannahJonah Holly HannahAndy JonahAndy Hannah Andy Hannah JonahHannahJonahHannah Jonah Holly Hannah JonahHolly JonahAndyJonah Andy Hannah Jonah Andy Hannah Jonah Holly Hannah Andy HannahHolly JonahAndy HannahJonah Holly HannahAndy JonahAndy Hannah Andy Hannah JonahHannahJonahHannah Jonah Holly Hannah JonahHolly JonahAndyJonah Andy Hannah Jonah Andy Hannah Jonah Holly Hannah Andy HannahHolly JonahAndy HannahJonah Holly HannahAndy JonahAndy

CVPR SLAM 2007 Classification with Group Prior From the joint pdf, inference questions can be answered: Most Probable Explanation MAP MAP- Most probable assignment of a particular person.

CVPR SLAM 2007 Experiment The image collection: Facial Features: Active Shape Model [Cootes95] based features, then PCA reduces to 5D. Images1197 Images with multiple people188 No. Faces in these images420 Individuals5 Ambiguously label a portion of the image collection, classify the identities of all the rest. Compare 4 Priors: Group Prior (GP) UOC Prior  A binary version of the GP that respects the UOC. The individual prior. No Prior The performance is quantified for: MAP MPE

CVPR SLAM 2007 Results Group Prior produces a large benefit. Note: All images were ambiguously labeled; no people were explicitly labeled. Example Classification (from 10 labeled images) Individual Prior / no PriorEthan Unique Object ConstraintHannahEthan Group PriorHollyEthan

CVPR SLAM 2007 Prior Work Many face recognition methods- most ignore the issue of prior probabilities. [Zhao03] Face recognition methods have been used to assist the labeling of image collections. [Zhang04] In news photos, names from captions have been assigned to faces. [Berg04] The co-occurrence of people in images has been studied, but not combined with image features. [Naaman05]

CVPR SLAM 2007 Conclusions The group prior models the social relationships between individuals. We learn feature distributions and relationships between the labels (people). By using the group prior, recognition accuracy is significantly improved!