Attributes for Classifier Feedback Amar Parkash and Devi Parikh

You can teach a child by examples...

And on, and on, and on…

Is this a giraffe?No. Is this a giraffe?Yes.Is this a giraffe?No.

And on, and on, and on…

Proposed Active Learning Scenario

I think this is a giraffe. What do you think? No, its neck is too short for it to be a giraffe. Ah! These must not be giraffes either then. [Animals with even shorter necks] …… Current belief Focused feedback Knowledge of the world Feedback on one, transferred to many Learner learns better from its mistakes Accelerated discriminative learning with few examples Learner learns better from its mistakes Accelerated discriminative learning with few examples

Communication Need a language that is Machine understandable Human understandable Attributes! Mid-level shareable Visual Semantic

Proposed Active Learning Concepts to teach: C1C1 C2C2 CKCK … Unlabeled pool of images Classifiers: h1h1 h2h2 hKhK … [Label-feedback] [Attributes-based feedback] Attribute predictors: a1a1 a2a2 aMaM … [Predicted Label] Any feature space Any discriminative learning algorithm

Relative Attributes [Parikh and Grauman, ICCV 2011] Openness Unlabeled pool of images Attribute predictors: a1a1 a2a2 aMaM … Image features Parameters

Attributes-based Feedback Unlabeled pool of images No, It is too open to be a forest Attribute predictors: a1a1 a2a2 aMaM … Forest Openness Not Forest

Attributes-based Feedback Unlabeled pool of images No, It is too open to be a forest Attribute predictors: a1a1 a2a2 aMaM … Forest Classifiers: h1h1 h2h2 hKhK … Not Forest

Proposed Active Learning Concepts to teach: C1C1 C2C2 CKCK … Unlabeled pool of images Classifiers: h1h1 h2h2 hKhK … [Label-feedback] [Attributes-based feedback] Attribute predictors: a1a1 a2a2 aMaM … [Predicted Label]

Label-based Feedback Not our contribution Experiment with different scenarios Benefits of attributes-based feedback – Small when label-based is very informative – Large when label-based feedback is weak

Unlabeled pool of images Forest Label-based Feedback Accept: Yes, this is a forest. – Strong: It is not anything else Example: Classification – Weak: It can be other things Example: Annotation

Label-based Feedback Reject: – Strong: No, it is a coast. Example: Classification with few classes – Weak: No, this is not a forest. Example: Large-scale classification Example: Biased binary classification 4 different scenarios in experiments Unlabeled pool of images Forest

Datasets Datasets and relative attribute predictors from [Parikh and Grauman, ICCV 2011]

Datasets Faces: – 8 celebrity categories – 11 attributes (chubby, white, etc.) [Kumar et al., ICCV 2009]

Datasets Scenes: – 8 categories – 6 attributes (open, natural, etc.) [Oliva and Torralba, IJCV 2001]

Settings Feedback from MTurk Features: – Raw image features (gist, color) – Attribute scores Category classifiers: SVM with RBF kernel Results on – 2 datasets x 2 features x 4 label-feedback scenarios – Show 2 here, rest in paper. This image doesnt have enough perspective to be a street scene.

Results Faces, Attribute features, Strong label-feedback # iterations Accuracy

Results Scenes, Image features, Weak label-feedback # iterations Accuracy 1/4 th ! More results in the paper.

Conclusion Attributes for providing classifier feedback Novel learning paradigm with enhanced human-machine communication Discriminative learning + domain knowledge Learning with few examples Connections to semi-supervised learning – Shrivastava, Singh and Gupta: Up Next!

Thank you!

Backup Slides

Negative Feedback Many reasons come together to make a concept – Hard to describe why an image is a concept One reason can break a concept – Easier to describe why an image is not a concept (Arguably) waste to give feedback when right

Discriminative – No domain knowledge is conveyed – Many training images – Discriminative model – Classification in any feature space – State-of-art performance Related Work Zero-shot learning – Convey domain knowledge – Zero training images – Generative model – Classification in attribute space – Performance compromised Proposed paradigm – Convey domain knowledge to transfer – Few training images – Discriminative model – Classification in any feature space – Performance maintained bea r turtlerabbit furry big [Lampert et al., CVPR 2009] C Smiling Age S J H C is younger than H C smiles more than H M M is younger than J M smiles more than J [Parikh and Grauman, ICCV 2011] – – –

Related Work Focused discrimination – Mining of hard negatives [Felzenszwalb 2010] – To understand classifier [Golland 2001] – Here, supervisor provides the discriminative direction by verbalizing semantic knowledge [Golland, NIPS 2001][Felzenszwalb et al., CVPR 2008]

Related Work What we are not doing: Collecting deeper annotations of images Segmentation masks [Russell 2008], Parts [Farhadi 2010], Pose [Bourdev 2009], Attributes [Kumar 2009] We use attributes for broad propagation of category labels to unlabeled images

Related Work What we are not doing: Actively interleaving attribute annotations Object & attributes [Kovashka 2011], Image, boxes & segments [Vijaynarsimhan 2008], Parts & attributes [Wah 2011], etc. Human-in-the-loop at test time [Branson 2010] Our supervisor provides additional information at training time which is leveraged for better category models

Related Work Rationales – Human feature selection in NLP [Raghavan 2005] – Spatial and attribute rationales [Donahue 2011] – Restricted to classification in attribute space – We can operate in any feature space [Donahue and Grauman, ICCV 2011]

Imperfect Attribute Predictors In the end, all images labeled with ground truth Discriminative training can deal with outliers Attributes are pre-trained, and so unlikely to be severely flawed Experiments: used predictors directly from Parikh and Grauman, 2011

Large-scale Classification Categories may require expert knowledge, attributes need not Can show a few exemplars to verify category

Results Faces, Classification, Attribute features (overestimate) # iterations Accuracy

Label-based Feedback ClassificationLarge-scale Classification Annotation Street Outdoor City Grayscale …. Biased Binary Classification Simulate the different scenarios in responses Benefits of attributes-based feedback Small when label-based is very informative Large when label-based feedback is weak Simulate the different scenarios in responses Benefits of attributes-based feedback Small when label-based is very informative Large when label-based feedback is weak

Results Scenes, Annotation, Image features # iterations Accuracy

Results Faces, Biased binary classification, Attribute features # iterations Accuracy More results in the paper.

Traditional Active Learning Concepts to teach: C1C1 C2C2 CKCK … Unlabeled pool of images Classifiers: h1h1 h2h2 hKhK … [Label] Highest Entropy