Relative Attributes Presenter: Shuai Zheng (Kyle) Supervised by Philip H.S. Torr Author: Devi Parikh (TTI-Chicago) and Kristen Grauman (UT-Austin)

What is visual attributes? Attributes are properties observable in images that have human-designated names, such as ‘Orange’, ‘striped’, or ‘Furry’.

Learning Binary Attributes In PASCAL VOC Challenge, we learn to predict binary attributes. (e.g., dog? Or not a dog?) Vittorio Ferrari, Andrew Zisserman. Learning Visual Attributes. NIPS O. Parkhi, A.Vedaldi C.V.Jawahar, A.Zisserman. The Truth About Cats and Dogs. ICCV 2011.

Problems within Binary Attributes Given an attribute it is easy to get labelled data on AMT(Amazon Mechanical Turk). But, where do attributes come from? Can we find a easier way to ask more people rather than experts to tag the images?

Problems within Binary Attributes Some tags are binary while some are relative. Is furry Has four-legs Has tail Tail longer than donkeys’ Legs shorter than horses’ Mule

Labeling data

What is relative attributes? Relative attribute indicates the strength of an attribute in an image with respect to other image rather than simply predicting the presence of an attribute.

Advantages of Relative Attributes Enhanced human-machine communication More informative Natural for humans Enhanced human-machine communication More informative Natural for humans 8

Contributions Propose a model to learn relative attributes –Allow relating images and categories to each other –Learn ranking function for each attribute Give two novel applications based on the model –Zero-shot learning from attribute comparisons –Automatically generating relative image Propose a model to learn relative attributes –Allow relating images and categories to each other –Learn ranking function for each attribute Give two novel applications based on the model –Zero-shot learning from attribute comparisons –Automatically generating relative image

Contributions Propose a model to learn relative attributes –Allow relating images and categories to each other –Learn ranking function for each attribute Give two novel applications based on the model –Zero-shot learning from attribute comparisons –Automatically generating relative image Propose a model to learn relative attributes –Allow relating images and categories to each other –Learn ranking function for each attribute Give two novel applications based on the model –Zero-shot learning from attribute comparisons –Automatically generating relative image

Learning Relative Attributes For each attribute Supervision is open

Learning Relative Attributes Learn a scoring function that best satisfies constraints: 12 Image features Learned parameters

Learning Relative Attributes 13 Max-margin learning to rank formulation Based on [Joachims 2002] Rank Margin Image Relative Attribute Score

Learning binary attributes v.s. Learning relative attributes

Contributions Propose a model to learn relative attributes –Allow relating images and categories to each other –Learn ranking function for each attribute Give two novel applications based on the model –Zero-shot learning from attribute comparisons –Automatically generating relative image Propose a model to learn relative attributes –Allow relating images and categories to each other –Learn ranking function for each attribute Give two novel applications based on the model –Zero-shot learning from attribute comparisons –Automatically generating relative image

Relative Zero-shot Learning Training: Images from S seen categories and Descriptions of U unseen categories Need not use all attributes, or all seen categories Testing: Categorize image into one of S+U categories 16 Age: Scarlett CliveHugh Jared Miley Smiling: Jared Miley

Relative Zero-shot Learning Clive Infer image category using max-likelihood Can predict new classes based on their relationships to existing classes – without training images 17 Age: Scarlett CliveHugh Jared Miley Smiling: Jared Miley Smiling Age Miley S J H

Contributions Propose a model to learn relative attributes –Allow relating images and categories to each other –Learn ranking function for each attribute Give two novel applications based on the model –Zero-shot learning from attribute comparisons –Automatically generating relative image Propose a model to learn relative attributes –Allow relating images and categories to each other –Learn ranking function for each attribute Give two novel applications based on the model –Zero-shot learning from attribute comparisons –Automatically generating relative image

Automatic Relative Image Description Density Conventional binary description: not dense Dense:Not dense: Novel image 19

more dense than less dense than Density Novel image 20 Automatic Relative Image Description

CCHHHCFHHMFFIF more dense than Highways, less dense than Forests Density Novel image 21 Automatic Relative Image Description

Contributions Propose a model to learn relative attributes –Allow relating images and categories to each other –Learn ranking function for each attribute Give two novel applications based on the model –Zero-shot learning from attribute comparisons –Automatically generating relative image Propose a model to learn relative attributes –Allow relating images and categories to each other –Learn ranking function for each attribute Give two novel applications based on the model –Zero-shot learning from attribute comparisons –Automatically generating relative image

Datasets Outdoor Scene Recognition (OSR) [Oliva 2001] 8 classes, ~2700 images, Gist 6 attributes: open, natural, etc. Public Figures Face (PubFig) [Kumar 2009] 8 classes, ~800 images, Gist+color 11 attributes: white, chubby, etc. 23 Attributes labeled at category level

Zero-shot learning – Binary attributes: Direct Attribute Prediction [Lampert 2009] – Relative attributes via classifier scores Automatic image-description – Binary attributes – – – Baselines bearturtlerabbit furry big

Relative Zero-shot Learning An attribute is more discriminative when used relatively Binary attributes Rel. att. (classifier) 25 Rel. att.(ranke r)

Relative (ours): More natural than insidecity Less natural than highway More open than street Less open than coast Has more perspective than highway Has less perspective than insidecity Binary (existing): Not natural Not open Has perspective Automatic Relative Image Description 26

Automatic Relative Image Description 18 subjects Test cases: 10OSR, 20 PubFig 27

Traditional Recognition DogChimpanzeeTiger ??? Tiger 28

Attributes-based Recognition Furry White Black Big Striped Yellow Striped Black White Big Attributes provide a mode of communication between humans and machines! [Lampert 2009] [Farhadi 2009] [Kumar 2009] [Berg 2010] [Parikh 2010] … Zero-shot learning Describing objects Face verification Attribute discovery Nameable attributes … 29 DogChimpanzeeTiger

Conclusions and Future Work Relative attributes learnt as ranking functions – Natural and accurate zero-shot learning of novel concepts by relating them to existing concepts – Precise image descriptions for human interpretation Attributes-based recognition is an interesting direction for the future object/scenes recognition. 30 Enhanced human-machine communication

Cheers! – Shuai Zheng (Kyle) Created by Tag clouds