Zeroshot Learning 2015.4.2 Mun Jonghwan

Zero-shot Learning Traindata : Which image shows a cat?

Zero-shot Learning Traindata : Which image shows a giraffe?

Zero-shot Learning Which image shows a giraffe? Description has long neck? Is black? Is spot? lives in plain?

External information Attribute Word vector Hierarchy Co-occurrence - C.H. Lampert, Attribute-based classification for zero-shot visual object classification, TPAMI13[1] D. Parikh, Relative attributes, ICCV2011[3] Z. Akata, Label embedding for attribute-based classification, CVPR13[2] Word vector - A. Frome, Devise: A deep visual-semantic embedding model, NIPS13[3] - Z. Akata, Evaluation of output embedding for fine-grained image classification, CVPR15[4] Hierarchy - Usually used as side information Co-occurrence - T. Mensink, Costa: Co-occurrence statistics for zero-shot classification, CVPR14

Direct Attribute Prediction (DAP)[1] Learn attribute classifier from related classes Use attribute-to-class mapping for prediction Label Attribute Image 𝑝 𝑎 𝑚 = 𝑎 𝑚 𝑧 𝑥 = 𝑝 𝑎 𝑚 𝑥 𝑖𝑓 𝑎 𝑚 𝑧 =1 1−𝑝 𝑎 𝑚 𝑥 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒 𝑧 ∗ = argmax 𝑧 𝑚 𝑝( 𝑎 𝑚 𝑧 |𝑥)

Direct Attribute Prediction (DAP)[1] 1. Vocabulary of attributes and class decriptions - giraffe has properties X and Y but not Z 2. Train classifier for each attribute X, Y, Z - From visual examples of related classes 3. Make image attributes predictions 4. Combine into decision: this image is not giraffe 𝑃 𝑋 𝑖𝑚𝑔 =0.8 𝑃 𝑌 𝑖𝑚𝑔 =0.3 𝑃 𝑍 𝑖𝑚𝑔 =0.7 ⇒

Relative Attribute[2] Problem : Binary attributes are very crude If mouse = small, then cat ≠ small If elephant = large, then cat ≠ large 𝑂 𝑚 : 𝑤 𝑚 𝑡 𝑥 𝑖 > 𝑤 𝑚 𝑡 𝑥 𝑗 𝑂 𝑚 : 𝑤 𝑚 𝑡 𝑥 𝑖 = 𝑤 𝑚 𝑡 𝑥 𝑗

Relative Attribute[2] S Clive Smiling J H Age Age: Scarlett Hugh Jared 1 2 ( 𝜇 𝐻 𝑠 + 𝜇 𝑆 𝑠 ) Clive Age: Scarlett Hugh Jared Miley Smiling: Smiling Age S J H 1 2 ( 𝜇 𝐽 𝑠 + 𝑑 𝑚 ) Infer image category using max-likelihood

Attribute Label Embedding (ALE)[2] Embedding to attribute space Search for the class with the highest compatibility

Word Vector[3] Use CNN feature Embedding vector is collected automatically from text corpora Embedding to word vector space

Word Vector[3] Semantically similar classes are close country capital Semantically similar classes are close Word relationship is represented as displacement - 𝐾𝑖𝑛𝑔 – 𝑀𝑎𝑛 + 𝑤𝑜𝑚𝑒𝑛 = 𝑄𝑢𝑒𝑒𝑛

Survey result[4]

Relative information from word vector Tiger : bobcat = strong : ? bobcat : tiger = small : ?

Relative information from word vector Some ranking information Attribute Attribute embedding

Thank you