CRCV REU 2019 Week 5
Current Approach Last Week: LSTM Investigate What LSTM misses Question + Answers vs Question + Answers with Video
Literature Review Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering Devi Parikh, CVPR 2017 Higher entropy in P(A|Q) such that image I must assist to determine A Explanation modality: counter-examples, close to but not belonging to category predicted by model
Results Total Q Total Q + V 6214 6221
Results Total Q Total Q + V Exclusive Q Exclusive Q + V 6214 6221 966 973
Results Total Q Total Q + V Exclusive Q Exclusive Q + V Both Missed 6214 6221 966 973 1442
Results Total Q Total Q + V Exclusive Q Exclusive Q + V Both Missed 6214 6221 966 973 1442 Who Total Q 647 Total Q + V 606
Results Total Q Total Q + V Exclusive Q Exclusive Q + V Both Missed 6214 6221 966 973 1442 Who Total Q 647 Total Q + V 606 Exclusive Q 164 Exclusive Q + V 124 Both Missed 286
Results Total Q Total Q + V Exclusive Q Exclusive Q + V Both Missed 6214 6221 966 973 1442 Who Where Total Q 647 721 Total Q + V 606 746 Exclusive Q 164 103 Exclusive Q + V 124 128 Both Missed 286 160
Results Total Q Total Q + V Exclusive Q Exclusive Q + V Both Missed 6214 6221 966 973 1442 Who Where What Total Q 647 721 3568 Total Q + V 606 746 3579 Exclusive Q 164 103 514 Exclusive Q + V 124 128 525 Both Missed 286 160 731
Results Total Q Total Q + V Exclusive Q Exclusive Q + V Both Missed 6214 6221 966 973 1442 Who Where What How Total Q 647 721 3568 566 Total Q + V 606 746 3579 580 Exclusive Q 164 103 514 70 Exclusive Q + V 124 128 525 84 Both Missed 286 160 731 136
Results Total Q Total Q + V Exclusive Q Exclusive Q + V Both Missed 6214 6221 966 973 1442 Who Where What How Why Total Q 647 721 3568 566 712 Total Q + V 606 746 3579 580 710 Exclusive Q 164 103 514 70 114 Exclusive Q + V 124 128 525 84 112 Both Missed 286 160 731 136 129
Next Steps Add Modules to the Baseline LSTM Further Explore the predictions