Zero shot learning Presented by: YuYing Chou email: d07922014@csie.ntu.edu.tw phone: 0928372603 Advisor: Tyng-Luh Liu, Hsuan-Tien Lin

What is zero shot learning Teach computer to recognize something they have not seen. How to make it become possible? https://applealmond.com/posts/28378

How human start to recognize something Imagine Guess with hint

what does alien look like

The zero shot task is similar … Baby tries to learn something new in the world. 我們要教machine 如何辨認斑馬

Give the machine hints C. H. Lampert, H. Nickisch, and S. Harmeling. "Learning To Detect Unseen Object Classes by Between-Class Attribute Transfer". In CVPR, 2009

Give the machine a hint The attributes are the hints given to the computer. C. H. Lampert, H. Nickisch, and S. Harmeling. "Learning To Detect Unseen Object Classes by Between-Class Attribute Transfer". In CVPR, 2009

Zebra wiki Zebras (/ˈziːbrə/ ZEE-brə, UK also /ˈzɛbrə/ ZEB-rə)[1] are several species of African equids (horse family) united by their distinctive black and white striped coats. Their stripes come in different patterns, unique to each individual. They are generally social animals that live in small harems to large herds. Unlike their closest relatives, horses and donkeys, zebras have never been truly domesticated. There are three species of zebras: the plains zebra, the mountain zebra and the Grévy's zebra. The plains zebra and the mountain zebra belong to the subgenus Hippotigris, but Grévy's zebra is the sole species of subgenus Dolichohippus. The latter resembles an ass, to which zebras are closely related, while the former two look more horse-like. All three belong to the genus Equus, along with other living equids. Frome, Andrea, Greg S. Corrado, Jon Shlens, Samy Bengio, Jeff Dean, and Tomas Mikolov. "Devise: A deep visual-semantic embedding model." In Advances in neural information processing systems, pp. 2121-2129. 2013.

Old models

From https://becominghuman.ai/back-propagation-in-convolutional-neural-networks-intuition-and-code-714ef1c38199 https://en.wikipedia.org/wiki/Generalised_logistic_function

The drawbacks of old models Zerba is similar to horses and has stripes Bias like horse spot

transductive learning Unseen data Unseen data Unsupervised learning

Fu, Yanwei, Timothy M. Hospedales, Tao Xiang, and Shaogang Gong Fu, Yanwei, Timothy M. Hospedales, Tao Xiang, and Shaogang Gong. "Transductive multi-view zero-shot learning." IEEE transactions on pattern analysis and machine intelligence 37, no. 11 (2015): 2332-2345.

Generative model to generate fake unseen image decoder

Generate Unseen image decoder

Use unseen image to train model (MLP: Multi-Level Perceptron)

𝐿𝑜𝑠𝑠= 𝑖=1 𝑁 [ 𝑓 𝑖 𝑎 − 𝑓 𝑖 𝑝 2 2 − 𝑓 𝑖 𝑎 − 𝑓 𝑖 𝑛 2 2 ] 𝑓 𝑖 𝑎 : a sample of image features ( 𝑓 𝑖 𝑎 , 𝑓 𝑖 𝑝 ) : a positive pair ( 𝑓 𝑖 𝑎 , 𝑓 𝑖 𝑛 ) : a negative pair Triplet pair 𝐿𝑜𝑠𝑠= 𝑖=1 𝑁 [ 𝑓 𝑖 𝑎 − 𝑓 𝑖 𝑝 2 2 − 𝑓 𝑖 𝑎 − 𝑓 𝑖 𝑛 2 2 ] 𝑓 𝑖 𝑎 𝑓 𝑖 𝑛 𝑓 𝑖 𝑝 𝑓 𝑖 𝑎 𝑓 𝑖 𝑛 𝑓 𝑖 𝑝 After learning

*method 1. CVAE (Conditional Variational Autoencoder) 2. MLP 3. Triplet loss

*steps 1. Generate seen classes features with CNN 2. Train CVAE with seen classes image features 3. Generate unseen classes image from CVAE 4. Train MLP with seen classes features and generated unseen classes image features

Caltech-UCSD Birds-200-2011

[2]Annadani, Yashas and Biswas, Soma Preserving Semantic Relations for Zero-Shot Learning ,arXiv preprint arXiv:1803.03049, 2018 [3] Chen, Long, Hanwang Zhang, Jun Xiao, Wei Liu, and Shih-Fu Chang. ”Zero-Shot Visual Recognition using Semantics-Preserving Adversarial Embedding Network.” arXiv preprint arXiv:1712.01928 (2017). [4] Arora, Gundeep, Vinay Kumar Verma, Ashish Mishra, and Piyush Rai. ”Generalized Zero-Shot Learning via Synthesized Examples.” arXiv preprint arXiv:1712.03878 (2017). [6] Song, Jie, Chengchao Shen, Yezhou Yang, Yang Liu, and Mingli Song. ”Transductive Unbiased Embedding for Zero-Shot Learning.” arXiv preprint arXiv:1803.11320 (2018).

*result The generative model very likely improves the ZSL task, but it will sacrifice the accuracy of training (seen) classes.