1. CRCV REU 2019
Week 5

2. Current Approach Last Week: LSTM Investigate What LSTM misses
Question + Answers vs Question + Answers with Video

3. Literature Review
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
Devi Parikh, CVPR 2017
Higher entropy in P(A|Q) such that image I must assist to determine A
Explanation modality: counter-examples, close to but not belonging to category predicted by model

4. Results
Total Q
Total Q + V
6214
6221

5. Results Total Q Total Q + V Exclusive Q Exclusive Q + V 6214 6221 966
973

6. Results Total Q Total Q + V Exclusive Q Exclusive Q + V Both Missed
6214
6221
966
973
1442

7. Results Total Q Total Q + V Exclusive Q Exclusive Q + V Both Missed
6214
6221
966
973
1442
Who
Total Q
647
Total Q + V
606

8. Results Total Q Total Q + V Exclusive Q Exclusive Q + V Both Missed
6214
6221
966
973
1442
Who
Total Q
647
Total Q + V
606
Exclusive Q
164
Exclusive Q + V
124
Both Missed
286

9. Results Total Q Total Q + V Exclusive Q Exclusive Q + V Both Missed
6214
6221
966
973
1442
Who
Where
Total Q
647
721
Total Q + V
606
746
Exclusive Q
164
103
Exclusive Q + V
124
128
Both Missed
286
160

10. Results Total Q Total Q + V Exclusive Q Exclusive Q + V Both Missed
6214
6221
966
973
1442
Who
Where
What
Total Q
647
721
3568
Total Q + V
606
746
3579
Exclusive Q
164
103
514
Exclusive Q + V
124
128
525
Both Missed
286
160
731

11. Results Total Q Total Q + V Exclusive Q Exclusive Q + V Both Missed
6214
6221
966
973
1442
Who
Where
What
How
Total Q
647
721
3568
566
Total Q + V
606
746
3579
580
Exclusive Q
164
103
514 70
Exclusive Q + V
124
128
525 84
Both Missed
286
160
731
136

12. Results Total Q Total Q + V Exclusive Q Exclusive Q + V Both Missed
6214
6221
966
973
1442
Who
Where
What
How
Why
Total Q
647
721
3568
566
712
Total Q + V
606
746
3579
580
710
Exclusive Q
164
103
514 70
114
Exclusive Q + V
124
128
525 84
112
Both Missed
286
160
731
136
129

13. Next Steps Add Modules to the Baseline LSTM
Further Explore the predictions