Describing People: A Poselet-Based Approach to Attribute Classification Lubomir Bourdev 1,2 Subhransu Maji 1 Jitendra Malik 1 1 EECS U.C. Berkeley 2 Adobe Systems Inc.

Goal: Extract attributes from images of people

Who has long hair?Who has long hair?

Who has short pants?Who has short pants?

Male or female?Male or female?

Prior work on poselets and on attributes

Prior work on PoseletsPrior work on Poselets Introduced by [Bourdev and Malik, ICCV09] Detection with poselets [Bourdev et al, ECCV10] Applications Segmentation [Brox et al, ECCV10] [Maire et al, ICCV 11] Actions [Yang et al, CVPR10] [Maji et al, CVPR11] [Yao et al, ICCV11] Human parsing [Wang et al, CVPR11] Semantic contours [Hariharan et al, ICCV11] Subordinate level categorization [Farrell et al, ICCV11]

Prior work on PoseletsPrior work on Poselets Introduced by [Bourdev and Malik, ICCV09] Detection with poselets [Bourdev et al, ECCV10] Applications Segmentation [Brox et al, ECCV10] [Maire et al, ICCV 11] Actions [Yang et al, CVPR10] [Maji et al, CVPR11] [Yao et al, ICCV11] Human parsing [Wang et al, CVPR11] Semantic contours [Hariharan et al, ICCV11] Subordinate level categorization [Farrell et al, ICCV11]

Prior work on PoseletsPrior work on Poselets Introduced by [Bourdev and Malik, ICCV09] Detection with poselets [Bourdev et al, ECCV10] Applications Segmentation [Brox et al, ECCV10] [Maire et al, ICCV 11] Actions [Yang et al, CVPR10] [Maji et al, CVPR11] [Yao et al, ICCV11] Human parsing [Wang et al, CVPR11] Semantic contours [Hariharan et al, ICCV11] Subordinate level categorization [Farrell et al, ICCV11]

Prior work on PoseletsPrior work on Poselets Introduced by [Bourdev and Malik, ICCV09] Detection with poselets [Bourdev et al, ECCV10] Applications Segmentation [Brox et al, ECCV10] [Maire et al, ICCV 11] Actions [Yang et al, CVPR10] [Maji et al, CVPR11] [Yao et al, ICCV11] Human parsing [Wang et al, CVPR11] Semantic contours [Hariharan et al, ICCV11] Subordinate level categorization [Farrell et al, ICCV11]

Prior work on PoseletsPrior work on Poselets Introduced by [Bourdev and Malik, ICCV09] Detection with poselets [Bourdev et al, ECCV10] Applications Segmentation [Brox et al, ECCV10] [Maire et al, ICCV 11] Actions [Yang et al, CVPR10] [Maji et al, CVPR11] [Yao et al, ICCV11] Human parsing [Wang et al, CVPR11] Semantic contours [Hariharan et al, ICCV11] Subordinate level categorization [Farrell et al, ICCV11]

Prior work on AttributesPrior work on Attributes Attributes as intermediate parts Discovering attributes from text Discovering attributes from images Attributes from motion capture Joint learning of classes & attributes Image retrieval with attributes Attributes and actions Active learning with attributes Attributes of people Gender attribute [Cottrell and Medcalfe, NIPS90] [Golomb et al, NIPS90] [Moghaddam & Yang, PAMI02] [Ferrari & Zisserman, NIPS07] [Kumar et al, ECCV08] [Gallagher and Chen, CVPR08] [Cao et al, ACM08] [Lampert et al, CVPR09] [Farhadi et al, CVPR 09] [Wang et al, BMVC09] [Wang and Forsyth, ICCV09] [Kumar et al, ICCV09] [Farhadi et al, CVPR10] [Berg et al, ECCV10] [Wang and Mori, ECCV10] [Sigal et al, ECCV10] [Branson el al, ECCV10] [Hwang et al, CVPR11] [Parikh and Grauman, CVPR11] [Douze et al, CVPR11] [Kovashka et al, ICCV11] [Liu et al, CVPR11] [Qiu et al, ICCV11] [Yao et al, ICCV11] [Dhar et al, CVPR11] [Parikh and Grauman, ICCV11] [Siddiquie et al, CVPR11]

Prior work on AttributesPrior work on Attributes Attributes as intermediate parts Discovering attributes from text Discovering attributes from images Attributes from motion capture Joint learning of classes & attributes Image retrieval with attributes Attributes and actions Active learning with attributes Attributes of people Gender attribute [Cottrell and Medcalfe, NIPS90] [Golomb et al, NIPS90] [Moghaddam & Yang, PAMI02] [Ferrari & Zisserman, NIPS07] [Kumar et al, ECCV08] [Gallagher and Chen, CVPR08] [Cao et al, ACM08] [Lampert et al, CVPR09] [Farhadi et al, CVPR 09] [Wang et al, BMVC09] [Wang and Forsyth, ICCV09] [Kumar et al, ICCV09] [Farhadi et al, CVPR10] [Berg et al, ECCV10] [Wang and Mori, ECCV10] [Sigal et al, ECCV10] [Branson el al, ECCV10] [Hwang et al, CVPR11] [Parikh and Grauman, CVPR11] [Douze et al, CVPR11] [Kovashka et al, ICCV11] [Liu et al, CVPR11] [Qiu et al, ICCV11] [Yao et al, ICCV11] [Dhar et al, CVPR11] [Parikh and Grauman, ICCV11] [Siddiquie et al, CVPR11]

Prior work on AttributesPrior work on Attributes Attributes as intermediate parts Discovering attributes from text Discovering attributes from images Attributes from motion capture Joint learning of classes & attributes Image retrieval with attributes Attributes and actions Active learning with attributes Attributes of people Gender attribute [Cottrell and Medcalfe, NIPS90] [Golomb et al, NIPS90] [Moghaddam & Yang, PAMI02] [Ferrari & Zisserman, NIPS07] [Kumar et al, ECCV08] [Gallagher and Chen, CVPR08] [Cao et al, ACM08] [Lampert et al, CVPR09] [Farhadi et al, CVPR 09] [Wang et al, BMVC09] [Wang and Forsyth, ICCV09] [Kumar et al, ICCV09] [Farhadi et al, CVPR10] [Berg et al, ECCV10] [Wang and Mori, ECCV10] [Sigal et al, ECCV10] [Branson el al, ECCV10] [Hwang et al, CVPR11] [Parikh and Grauman, CVPR11] [Douze et al, CVPR11] [Kovashka et al, ICCV11] [Liu et al, CVPR11] [Qiu et al, ICCV11] [Yao et al, ICCV11] [Dhar et al, CVPR11] [Parikh and Grauman, ICCV11] [Siddiquie et al, CVPR11]

Prior work on AttributesPrior work on Attributes Attributes as intermediate parts Discovering attributes from text Discovering attributes from images Attributes from motion capture Joint learning of classes & attributes Image retrieval with attributes Attributes and actions Active learning with attributes Attributes of people Gender attribute [Cottrell and Medcalfe, NIPS90] [Golomb et al, NIPS90] [Moghaddam & Yang, PAMI02] [Ferrari & Zisserman, NIPS07] [Kumar et al, ECCV08] [Gallagher and Chen, CVPR08] [Cao et al, ACM08] [Lampert et al, CVPR09] [Farhadi et al, CVPR 09] [Wang et al, BMVC09] [Wang and Forsyth, ICCV09] [Kumar et al, ICCV09] [Farhadi et al, CVPR10] [Berg et al, ECCV10] [Wang and Mori, ECCV10] [Sigal et al, ECCV10] [Branson el al, ECCV10] [Hwang et al, CVPR11] [Parikh and Grauman, CVPR11] [Douze et al, CVPR11] [Kovashka et al, ICCV11] [Liu et al, CVPR11] [Qiu et al, ICCV11] [Yao et al, ICCV11] [Dhar et al, CVPR11] [Parikh and Grauman, ICCV11] [Siddiquie et al, CVPR11]

Prior work on AttributesPrior work on Attributes Attributes as intermediate parts Discovering attributes from text Discovering attributes from images Attributes from motion capture Joint learning of classes & attributes Image retrieval with attributes Attributes and actions Active learning with attributes Attributes of people Gender attribute [Cottrell and Medcalfe, NIPS90] [Golomb et al, NIPS90] [Moghaddam & Yang, PAMI02] [Ferrari & Zisserman, NIPS07] [Kumar et al, ECCV08] [Gallagher and Chen, CVPR08] [Cao et al, ACM08] [Lampert et al, CVPR09] [Farhadi et al, CVPR 09] [Wang et al, BMVC09] [Wang and Forsyth, ICCV09] [Kumar et al, ICCV09] [Farhadi et al, CVPR10] [Berg et al, ECCV10] [Wang and Mori, ECCV10] [Sigal et al, ECCV10] [Branson el al, ECCV10] [Hwang et al, CVPR11] [Parikh and Grauman, CVPR11] [Douze et al, CVPR11] [Kovashka et al, ICCV11] [Liu et al, CVPR11] [Qiu et al, ICCV11] [Yao et al, ICCV11] [Dhar et al, CVPR11] [Parikh and Grauman, ICCV11] [Siddiquie et al, CVPR11]

Prior work on AttributesPrior work on Attributes Attributes as intermediate parts Discovering attributes from text Discovering attributes from images Attributes from motion capture Joint learning of classes & attributes Image retrieval with attributes Attributes and actions Active learning with attributes Attributes of people Gender attribute [Cottrell and Medcalfe, NIPS90] [Golomb et al, NIPS90] [Moghaddam & Yang, PAMI02] [Ferrari & Zisserman, NIPS07] [Kumar et al, ECCV08] [Gallagher and Chen, CVPR08] [Cao et al, ACM08] [Lampert et al, CVPR09] [Farhadi et al, CVPR 09] [Wang et al, BMVC09] [Wang and Forsyth, ICCV09] [Kumar et al, ICCV09] [Farhadi et al, CVPR10] [Berg et al, ECCV10] [Wang and Mori, ECCV10] [Sigal et al, ECCV10] [Branson el al, ECCV10] [Hwang et al, CVPR11] [Parikh and Grauman, CVPR11] [Douze et al, CVPR11] [Kovashka et al, ICCV11] [Liu et al, CVPR11] [Qiu et al, ICCV11] [Yao et al, ICCV11] [Dhar et al, CVPR11] [Parikh and Grauman, ICCV11] [Siddiquie et al, CVPR11]

Prior work on AttributesPrior work on Attributes Attributes as intermediate parts Discovering attributes from text Discovering attributes from images Attributes from motion capture Joint learning of classes & attributes Image retrieval with attributes Attributes and actions Active learning with attributes Attributes of people Gender attribute [Cottrell and Medcalfe, NIPS90] [Golomb et al, NIPS90] [Moghaddam & Yang, PAMI02] [Ferrari & Zisserman, NIPS07] [Kumar et al, ECCV08] [Gallagher and Chen, CVPR08] [Cao et al, ACM08] [Lampert et al, CVPR09] [Farhadi et al, CVPR 09] [Wang et al, BMVC09] [Wang and Forsyth, ICCV09] [Kumar et al, ICCV09] [Farhadi et al, CVPR10] [Berg et al, ECCV10] [Wang and Mori, ECCV10] [Sigal et al, ECCV10] [Branson el al, ECCV10] [Hwang et al, CVPR11] [Parikh and Grauman, CVPR11] [Douze et al, CVPR11] [Kovashka et al, ICCV11] [Liu et al, CVPR11] [Qiu et al, ICCV11] [Yao et al, ICCV11] [Dhar et al, CVPR11] [Parikh and Grauman, ICCV11] [Siddiquie et al, CVPR11]

Prior work on AttributesPrior work on Attributes Attributes as intermediate parts Discovering attributes from text Discovering attributes from images Attributes from motion capture Joint learning of classes & attributes Image retrieval with attributes Attributes and actions Active learning with attributes Attributes of people Gender attribute [Cottrell and Medcalfe, NIPS90] [Golomb et al, NIPS90] [Moghaddam & Yang, PAMI02] [Ferrari & Zisserman, NIPS07] [Kumar et al, ECCV08] [Gallagher and Chen, CVPR08] [Cao et al, ACM08] [Lampert et al, CVPR09] [Farhadi et al, CVPR 09] [Wang et al, BMVC09] [Wang and Forsyth, ICCV09] [Kumar et al, ICCV09] [Farhadi et al, CVPR10] [Berg et al, ECCV10] [Wang and Mori, ECCV10] [Sigal et al, ECCV10] [Branson el al, ECCV10] [Hwang et al, CVPR11] [Parikh and Grauman, CVPR11] [Douze et al, CVPR11] [Kovashka et al, ICCV11] [Liu et al, CVPR11] [Qiu et al, ICCV11] [Yao et al, ICCV11] [Dhar et al, CVPR11] [Parikh and Grauman, ICCV11] [Siddiquie et al, CVPR11]

Prior work on AttributesPrior work on Attributes Attributes as intermediate parts Discovering attributes from text Discovering attributes from images Attributes from motion capture Joint learning of classes & attributes Image retrieval with attributes Attributes and actions Active learning with attributes Attributes of people Gender attribute [Cottrell and Medcalfe, NIPS90] [Golomb et al, NIPS90] [Moghaddam & Yang, PAMI02] [Ferrari & Zisserman, NIPS07] [Kumar et al, ECCV08] [Gallagher and Chen, CVPR08] [Cao et al, ACM08] [Lampert et al, CVPR09] [Farhadi et al, CVPR 09] [Wang et al, BMVC09] [Wang and Forsyth, ICCV09] [Kumar et al, ICCV09] [Farhadi et al, CVPR10] [Berg et al, ECCV10] [Wang and Mori, ECCV10] [Sigal et al, ECCV10] [Branson el al, ECCV10] [Hwang et al, CVPR11] [Parikh and Grauman, CVPR11] [Douze et al, CVPR11] [Kovashka et al, ICCV11] [Liu et al, CVPR11] [Qiu et al, ICCV11] [Yao et al, ICCV11] [Dhar et al, CVPR11] [Parikh and Grauman, ICCV11] [Siddiquie et al, CVPR11]

Prior work on AttributesPrior work on Attributes Attributes as intermediate parts Discovering attributes from text Discovering attributes from images Attributes from motion capture Joint learning of classes & attributes Image retrieval with attributes Attributes and actions Active learning with attributes Attributes of people Gender attribute [Cottrell and Medcalfe, NIPS90] [Golomb et al, NIPS90] [Moghaddam & Yang, PAMI02] [Ferrari & Zisserman, NIPS07] [Kumar et al, ECCV08] [Gallagher and Chen, CVPR08] [Cao et al, ACM08] [Lampert et al, CVPR09] [Farhadi et al, CVPR 09] [Wang et al, BMVC09] [Wang and Forsyth, ICCV09] [Kumar et al, ICCV09] [Farhadi et al, CVPR10] [Berg et al, ECCV10] [Wang and Mori, ECCV10] [Sigal et al, ECCV10] [Branson el al, ECCV10] [Hwang et al, CVPR11] [Parikh and Grauman, CVPR11] [Douze et al, CVPR11] [Kovashka et al, ICCV11] [Liu et al, CVPR11] [Qiu et al, ICCV11] [Yao et al, ICCV11] [Dhar et al, CVPR11] [Parikh and Grauman, ICCV11] [Siddiquie et al, CVPR11]

Prior work on AttributesPrior work on Attributes Attributes as intermediate parts Discovering attributes from text Discovering attributes from images Attributes from motion capture Joint learning of classes & attributes Image retrieval with attributes Attributes and actions Active learning with attributes Attributes of people Gender attribute [Cottrell and Medcalfe, NIPS90] [Golomb et al, NIPS90] [Moghaddam & Yang, PAMI02] [Ferrari & Zisserman, NIPS07] [Kumar et al, ECCV08] [Gallagher and Chen, CVPR08] [Cao et al, ACM08] [Lampert et al, CVPR09] [Farhadi et al, CVPR 09] [Wang et al, BMVC09] [Wang and Forsyth, ICCV09] [Kumar et al, ICCV09] [Farhadi et al, CVPR10] [Berg et al, ECCV10] [Wang and Mori, ECCV10] [Sigal et al, ECCV10] [Branson el al, ECCV10] [Hwang et al, CVPR11] [Parikh and Grauman, CVPR11] [Douze et al, CVPR11] [Kovashka et al, ICCV11] [Liu et al, CVPR11] [Qiu et al, ICCV11] [Yao et al, ICCV11] [Dhar et al, CVPR11] [Parikh and Grauman, ICCV11] [Siddiquie et al, CVPR11]

Poselets for Attribute Classification

Male or female?Male or female?

Gender recognition is easier if we factor out the pose

Poselets [Bourdev & Malik ICCV09][Bourdev & Malik ICCV09]

Poselets Examples may differ visually but have common semantics

How do we train a poselet?How do we train a poselet?

Finding correspondences at training timeFinding correspondences at training time Given part of a human pose How do we find a similar pose configuration in the training set?

We use keypoints to annotate the joints, eyes, nose, etc. of people Left Hip Left Shoulder Finding correspondences at training timeFinding correspondences at training time

Residual Error Finding correspondences at training timeFinding correspondences at training time

Training poselet classifiersTraining poselet classifiers Residual Error: Given a seed patch 2. Find the closest patch for every other person 3. Sort them by residual error 4. Threshold them

Training poselet classifiersTraining poselet classifiers 1. Given a seed patch 2. Find the closest patch for every other person 3. Sort them by residual error 4. Threshold them 5. Use them as positive training examples to train a linear SVM with HOG features

Attribute Classification Algorithm at Test Time

Goal: Extract attributes of this person

Target person bounds Bounds of other nearby people Input:

Step 1: Detect poselet activations [Bourdev et al, ECCV10]

Step 2: Cluster the activations [Bourdev et al, ECCV10]

Step 3: Predict person bounds [Bourdev et al, ECCV10]

Step 4: Identify the correct cluster Max-flow in bipartite graph

Poselet Activations Start with its poselet activationsStart with its poselet activations

Features Features Poselet Activations Pyramid HOG LAB histogram Skin features Hands-skin Legs-skin Poselet patch B.* C Skin mask Arms mask

Poselet Activations Features Poselet-level Attribute Classifiers Attribute Classification OverviewAttribute Classification Overview

Poselet Activations Features Poselet-level Attribute Classifiers Person-level Attribute Classifiers Attribute Classification OverviewAttribute Classification Overview

Poselet Activations Features Poselet-level Attribute Classifiers Person-level Attribute Classifiers Context-level Attribute Classifiers Attribute Classification OverviewAttribute Classification Overview

Results

Our datasetOur dataset Source: VOC 2010 trainval for Person + H3D ~8000 annotations (4000 train test) 9 binary attributes specified by 5 independent annotators via AMT Ground truth label: If 4 of the 5 agree Dataset will be made publicly available

Visual search on our test setVisual search on our test set “Female” “Wears hat”

“Has long hair” “Wears glasses”

“Wears shorts” “Has long sleeves”

“Doesn’t have long sleeves”

Our baselineOur baseline Canny-modulated HOG with SPM kernel [Lazebnik et al CVPR06] To help the baseline trained separate SPM for four viewpoints: For each attribute we pick the best SPM as our baseline Full viewHead zoomUpper bodyLegs

Precision/recall on our test setPrecision/recall on our test set Label frequency - - ___ SPM ___ No context ___ Full Model

State-of-the-art Gender RecognitionState-of-the-art Gender Recognition We outperform Cognitec (top-notch face recognizer) We outperform any gender recognizer based on frontal faces (are there others?) 61% of our test have frontal faces. Even with perfect classification of frontal faces, max AP=80.5% vs. our AP of 82.4%

Men most confused as womenConfusions Women most confused as men baseball hatlong hairhair hidden

Short pants most confused to be long pants Non-T-shirt most confused to be T-shirt annotation errors Are these pants short?wrong person occlusion

Best poselets per attributeBest poselets per attribute Gender: Long Hair: Wears glasses:

“A woman with long hair, glasses and long pants”(??) We can describe a picture of a personWe can describe a picture of a person

Conclusion

How poselets help in high-level visionHow poselets help in high-level vision The image is a complex function of the viewpoint, pose, appearance, etc. Poselets decouple pose and camera view from appearance

Google “poselets” to get:Google “poselets” to get: The set of published poselet papers H3D data set + Matlab tools Java3D annotation tool + video tutorial Matlab code to detect people using poselets Our latest trained poselets

“A man with short hair, glasses, short sleeves and shorts” “A man with short hair and long sleeves” “A person with short hair, no hat and long sleeves” “A woman with long hair, glasses, short sleeves and long pants” “A person with long pants” Describing peopleDescribing people “A computer vision professor who likes machine learning” Failure mode Poselets websitePoselets website The set of published poselet papers H3D data set + Matlab tools Java3D annotation tool + video tutorial Matlab code to detect people using poselets Our latest trained poselets