1. Image Question Answering
Computer Vision Lab.
Paul Hongsuck Seo

2. Taxonomy of Sub-problems by hyeonwoo
Classification with Complex Setting.
Multi-domain classification
Classification with input/output connection
Zero-shot learning
Novel Computer Vision Tasks
Reference problem
Spatial relation problem
Visual semantic role labeling [6]
Weakly-supervised learning to count
Data Efficiency Problem
Operation compositionality
Image QA task compositionality
Natural Language Understanding
Extracting operation and input from question
Multi-modal Information Mergence
Merging multi-modal information better
Image Question Answering

3. Reference Problem
Different answers are correct given different targets of questions
what color is the cup?
What color is the teapot?
What color is the spoon?
Image Question Answering

4. Spatial Relation Understanding
Targets can be specified implicitly by their spatial relation to others.
What is behind the horse?
What is in front of the bed?
What is beside the cat?
Image Question Answering

5. General Idea – Attention Model
Attention model in caption generation
standing is
man
<BOS>
<EOS>
Attention
<INIT>
Then, Say
Attend First
Image Question Answering

6. General Idea – Attention Model
Reverse approach for IQA
yes
Then, Attend
Attention
Attention
Attention
Attention
Listen
First
<INIT> is
man
standing
<EOS>
Image Question Answering

7. MNIST-Reference Dataset
A synthetic dataset for the reference problem made of the MNIST dataset
Predicting the color given a number in an image
Image with one-hot information
Only partial information available from the answers
The objective function does not explicitly model the use of extra information
Image Question Answering

8. Multi-modal Information Mergence
In some tasks, answer classes don’t include any information about the ask
Example: yes/no question
Answering the existence of a given number in an image + =
number: 3
yes + =
number: 6 no
Image Question Answering

9. Models concat emb gating cgating CNN (flatten) CNN (fc out)
pool (2, 2)
conv (20, 3, 3)
flatten ()
one_hot (10)
concat ()
linear (200)
softmax (2)
CNN (fc out)
pool (2, 2)
conv (20, 3, 3)
fc (200)
one_hot (10)
emb (200)
nonlinearity ()
el-mul ()
linear (256)
softmax (2)
pool (2, 2)
conv (20, 3, 3)
fc (200)
one_hot (10)
emb (200)
concat ()
linear (256)
softmax (2)
CNN (fc out)
CNN
pool (2, 2)
conv (20, 3, 3)
one_hot (10)
emb (20)
nonlinearity ()
fc (200)
cgating ()
linear (256)
softmax (2)
Image Question Answering

10. Convolutional Gating
Individual pixel-wise gating on channels in a 2D feature map
Spatial feature selection
Image Question Answering

11. Results concat emb gating cgating linear tanh sigmoid Linear 60000
58.52
58.75
64.24
63.56
60.30
62.44
64.03
61.66
200000
79.19
77.21
85.73
85.91
81.15
90.80
89.30
81.69
gating_sigmoid
small var
large var
60000
58.49
59.98
200000
58.13
81.15
cgating_sigmoid
small var
large var
60000
58.40
61.66
200000
58.33
81.69
Image Question Answering

12. Discussion
Gating (element-wise multiplication) aligns the semantic arrangement in two modalities.
Values in each dimension of two vectors are semantically aligned.
The multiplication of the original feature representation performed better than the output of nonlinear functions.
Linear multiplication outperformed in both gating & cgating settings
The performances with gating showed high dependency on the variance of initial values.
Different variances performed better on different problems, different gatings, and different dataset sizes.
Image Question Answering

13. More Discussion + = + = + =
Sigmoid performed better in the real set (done by hyeonwoo)
Without yes/no questions
To test on the set with only yes/no questions
The task might be used for pre-training spatial transformer network. + =
number: 3
yes + =
number: 6 no + =
What is the animal?
elephant
Image Question Answering

14. Back to Reference Problem
Attention Model
Previous models were connected as the localization network of the attention model
CNN +
Localization
Network
number: 3 = 3
yellow
Image Question Answering

15. Spatial Transformer Network
Image Question Answering

16. Results baseline (concat) concat emb gating cgating linear tanh
sigmoid
Linear
60000
43.13
43.15
43.77
43.96
43.57
43.71
43.73
200000
65.25　
91.13
92.78
81.50
90.12
87.10
Image Question Answering

17. To Do Fill the blanks in the results
Comparisons between different parameters for gating models
Comparisons on larger networks (for cgating model)
Applying to the recurrent attention reasoning model with LSTM (or GRU)
Image Question Answering