Large-Scale Object Recognition using Label Relation Graphs Jia Deng 1,2, Nan Ding 2, Yangqing Jia 2, Andrea Frome 2, Kevin Murphy 2, Samy Bengio 2, Yuan Li 2, Hartmut Neven 2, Hartwig Adam 2 University of Michigan 1, Google 2

Object Classification Assign semantic labels to objects Corgi Puppy Dog Cat ✔ ✔ ✖ ✔

Object Classification Assign semantic labels to objects Probabilities Corgi Puppy Dog Cat

Object Classification Assign semantic labels to objects Feature Extractor FeaturesClassifier Probabilities Corgi Puppy Dog Cat

Object Classification Multiclass classifier: Softmax Corgi Puppy Dog Cat / / / / + Assumes mutual exclusive labels Independent binary classifiers: Logistic Regression Corgi Puppy Dog Cat No assumptions about relations.

Object labels have rich relations Corgi Puppy Dog Cat Exclusion Hierarchical Dog Cat Corg i Puppy Overlap Softmax: all labels are mutually exclusive  Logistic Regression: all labels overlap 

Goal: A new classification model Respects real world label relations Corgi Puppy Dog Cat Corgi Puppy Dog Cat

Visual Model + Knowledge Graph Corgi Puppy Dog Cat Visual Model Knowledge Graph Joint Inference Assumption in this work: Knowledge graph is given and fixed.

Agenda Encoding prior knowledge (HEX graph) Classification model Efficient Exact Inference Experiments Conclusion and Future Work

Agenda Encoding prior knowledge (HEX graph) Classification model Efficient Exact Inference Experiments Conclusion and Future Work

Hierarchy and Exclusion (HEX) Graph Corgi Puppy Dog Cat Exclusion Hierarchical Hierarchical edges (directed) Exclusion edges (undirected)

Examples of HEX graphs Car Bird Dog Cat Male Female Person Child Boy Round Red Shiny Thick Mutually exclusiveAll overlapping Combination Girl

State Space: Legal label configurations DogCatCorgiPuppy … … Corgi Puppy Dog Cat Each edge defines a constraint.

State Space: Legal label configurations Corgi Puppy Dog Cat DogCatCorgiPuppy … … Hierarchy: (dog, corgi) can’t be (0,1) Each edge defines a constraint.

State Space: Legal label configurations Corgi Puppy Dog Cat DogCatCorgiPuppy … … Exclusion: (dog, cat) can’t be (1,1) Hierarchy: (dog, corgi) can’t be (0,1) Each edge defines a constraint.

Agenda Encoding prior knowledge (HEX graph) Classification model Efficient Exact Inference Experiments Conclusion and Future Work

HEX Classification Model Pairwise Conditional Random Field (CRF) Input scores Binary Label vector

HEX Classification Model Pairwise Conditional Random Field (CRF) Binary Label vector Unary: same as logistic regression Input scores

HEX Classification Model Pairwise Conditional Random Field (CRF) Binary Label vector  All illegal configurations have probability zero. Unary: same as logistic regression If violates constraints Otherwise 0 Pairwise: set illegal configuration to zero Input scores

HEX Classification Model Pairwise Conditional Random Field (CRF) Binary Label vector Partition function: Sum over all (legal) configurations Input scores

HEX Classification Model Pairwise Conditional Random Field (CRF) Binary Label vector Probability of a single label: marginalize all other labels. Input scores

Special Case of HEX Model Softmax Car Bird Dog Cat Round Red Shiny Mutually exclusive All overlapping Thick Logistic Regressions

Learning Corgi Puppy Dog Cat DNN Label: Dog Maximize marginal probability of observed labels Back Propagation Dog Corgi Puppy Cat 1 ? ? ?

Agenda Encoding prior knowledge (HEX graph) Classification model Efficient Exact Inference Experiments Conclusion

Naïve Exact Inference is Intractable Inference: – Computing partition function – Perform marginalization HEX-CRF can be densely connected (large treewidth)

Observation 1: Exclusions are good Car Bird Dog Cat Lots of exclusions  Small state space  Efficient inference Realistic graphs have lots of exclusions. Rigorous analysis in paper. Number of legal states is O(n), not O(2 n ).

Observation 2: Equivalent graphs Dog Cat Corgi Puppy Pembroke Welsh Corgi Cardigan Welsh Corgi Dog Cat Corgi Puppy Pembroke Welsh Corgi Cardigan Welsh Corgi

Observation 2: Equivalent graphs Sparse equivalent Small Treewidth Dynamic programming Dog Cat Corgi Puppy Pembroke Welsh Corgi Cardigan Welsh Corgi Dog Cat Corgi Puppy Pembroke Welsh Corgi Cardigan Welsh Corgi Dog Cat Corgi Puppy Pembroke Welsh Corgi Cardigan Welsh Corgi Dense equivalent Prune states Can brute force

A B F B E D C G F B C F HEX Graph Inference A B E D C G F A B E D C G F A B E D C G F A B F B E D C G F B C F 1. Sparsify (offline) 3.Densify (offline) 2.Build Junction Tree (offline) 4.Prune Clique States (offline) 5. Message Passing on legal states (online)

Agenda Encoding prior knowledge (HEX graph) Classification model Efficient Exact Inference Experiments Conclusion and Future Work

Exp 1: Learning with weak labels Many basic category labels Few fine-grained labels Dog Corgi Animal … … Weak labels: No information on subcategories.

Corgi Puppy Dog Cat DNN Label: Dog Dog Corgi Puppy Cat 1 ? ? ? Hypothesis: HEX models can improve fine-grained recognition using basic level labels. Exp 1: Learning with weak labels

ILSVRC 2012: “relabel” or “weaken” a portion of fine-grained leaf labels to basic level labels. Evaluate on fine-grained recognition Exp 1: Learning with weak labels Dog Corgi Animal … Husky Dog Corgi Animal … Husky Relabel Training (“weakened” labels) Test Dog Corgi Animal … Husky Original ILSVRC 2012 (leaf labels)

ILSVRC 2012: “relabel” or “weaken” a portion of fine-grained leaf labels to basic level labels. Evaluate on fine-grained recognition. Consistently outperforms baselines. Exp 1: Learning with weak labels Top 1 accuracy (top 5 accuracy)

Exp 2: Zero-Shot Recognition using Object-Attribute Knowledge Animals with Attribute (AwA) dataset (Lampert et al. 2009) Training: Observe only a subset of animal labels. Given all animal-attribute relations Indirectly learns attributes. Test: predict new classes with no images in training. DAP (Lampert et al.)IAP (Lampert et al.)Ours 40.5%27.8%38.5% polar bear black: no white: yes brown: no stripes: no polar bear black white zebra brown stripes zebra black: yes white: yes brown: no stripes: yes … …

Related Work Multilabel Annotation & Hierarchy [Lampert et al. NIPS’11] [Chen et al. ICCV’11] [Bi & Kwok, NIPS’12] [Bucak et al. CVPR’11] Ours: Unifies hierarchy and exclusion. Transfer learning & Attributes [Rohrbach et al. CVPR’10] [Lampert et al. CVPR’09] [Kuettel et al. ECCV’12] [Akata et al. CVPR’13] Ours: A classification model that allows transferring. Extracting Common Sense Knowledge [Chen et al. ICCV’13] [Zitnick & Parikh CVPR’13] Ours: Assumes knowledge is given. [Hwang et al. CVPR’11] [Kang et al. CVPR’06] [Marszalek & Schmid CVPR’07] [Zweig & Weinshall CVPR’07] [Farhadi et al. CVPR’10] [Lim et al. NIPS’11] [Yu et al. CVPR’13] [Fergus et al. ECCV’10] [Zhu et al. ECCV’14] [Fouhey & Zitnick CVPR’14] Visual Model External Knowledge

Conclusions A unified framework for single object classification – Generalizes standard classification models – Leverages a knowledge graph – Efficient exact inference Future work – Non-absolute relations – Spatial relations between object instances