1. Zeroshot Learning
Mun Jonghwan

2. Zero-shot Learning
Traindata :
Which image shows a cat?

3. Zero-shot Learning
Traindata :
Which image shows a giraffe?

4. Zero-shot Learning Which image shows a giraffe? Description
has long neck?
Is black?
Is spot?
lives in plain?

5. External information Attribute Word vector Hierarchy Co-occurrence
- C.H. Lampert, Attribute-based classification for zero-shot visual object classification, TPAMI13[1]
D. Parikh, Relative attributes, ICCV2011[3]
Z. Akata, Label embedding for attribute-based classification, CVPR13[2]
Word vector
- A. Frome, Devise: A deep visual-semantic embedding model, NIPS13[3]
- Z. Akata, Evaluation of output embedding for fine-grained image classification, CVPR15[4]
Hierarchy
- Usually used as side information
Co-occurrence
- T. Mensink, Costa: Co-occurrence statistics for zero-shot classification, CVPR14

6. Direct Attribute Prediction (DAP)[1]
Learn attribute classifier from related classes
Use attribute-to-class mapping for prediction
Label
Attribute
Image
𝑝 𝑎 𝑚 = 𝑎 𝑚 𝑧 𝑥 = 𝑝 𝑎 𝑚 𝑥 𝑖𝑓 𝑎 𝑚 𝑧 =1 1−𝑝 𝑎 𝑚 𝑥 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
𝑧 ∗ = argmax 𝑧 𝑚 𝑝( 𝑎 𝑚 𝑧 |𝑥)

7. Direct Attribute Prediction (DAP)[1]
1. Vocabulary of attributes and class decriptions
- giraffe has properties X and Y but not Z
2. Train classifier for each attribute X, Y, Z
- From visual examples of related classes
3. Make image attributes predictions
4. Combine into decision: this image is not giraffe
𝑃 𝑋 𝑖𝑚𝑔 =0.8
𝑃 𝑌 𝑖𝑚𝑔 =0.3
𝑃 𝑍 𝑖𝑚𝑔 =0.7 ⇒

8. Relative Attribute[2] Problem : Binary attributes are very crude
If mouse = small, then cat ≠ small
If elephant = large, then cat ≠ large
𝑂 𝑚 : 𝑤 𝑚 𝑡 𝑥 𝑖 > 𝑤 𝑚 𝑡 𝑥 𝑗
𝑂 𝑚 : 𝑤 𝑚 𝑡 𝑥 𝑖 = 𝑤 𝑚 𝑡 𝑥 𝑗

9. Relative Attribute[2] S Clive Smiling J H Age Age: Scarlett Hugh Jared
1 2 ( 𝜇 𝐻 𝑠 + 𝜇 𝑆 𝑠 )
Clive
Age:
Scarlett
Hugh
Jared
Miley
Smiling:
Smiling
Age S J H
1 2 ( 𝜇 𝐽 𝑠 + 𝑑 𝑚 )
Infer image category using max-likelihood

10. Attribute Label Embedding (ALE)[2]
Embedding to attribute space
Search for the class with the highest compatibility

11. Word Vector[3] Use CNN feature
Embedding vector is collected automatically from text corpora
Embedding to word vector space

12. Word Vector[3] Semantically similar classes are close
country
capital
Semantically similar classes are close
Word relationship is represented as displacement
- 𝐾𝑖𝑛𝑔 – 𝑀𝑎𝑛 + 𝑤𝑜𝑚𝑒𝑛 = 𝑄𝑢𝑒𝑒𝑛

13. Survey result[4]

14. Relative information from word vector
Tiger : bobcat = strong : ?
bobcat : tiger = small : ?

15. Relative information from word vector
Some ranking information
Attribute
Attribute
embedding

16. Thank you