VQA: Visual Question Answering Presented by: Vivek Pradhan

Problem Definition Given an image we want to answer an open ended question about the image.

Best Performing Model Question and image encodings are fed to a LSTM to generate the answer. Scene level and object level information is available since it works well on these questions.

Experiments Why don’t humans have perfect accuracy? What does the model really learn?

Human Performance Humans do not do a perfect job. Captions give a lot of information. The model does not outperform humans with captions. It will be interesting to see if the set of answers that humans get right with captions significantly overlap with the model’s correct answers.

Annotator Accuracy

Annotator Accuracy

Annotator Accuracy

Annotator Accuracy

Naïve Baselines - Performance Method Accuracy Most Popular answer by question type 36.18% Answer chosen by answer to NN 40.61% Clearly it is possible to learn some shortcuts to solve this task. Captions Co-Occurrence Likely Answer by Scene Category Likely Answer by Object Detection

Possible Extensions Will the addition of object labels and scene categories help? Can we use a object detection model and scene classification model to provide better signals than VGG?

What does the model learn?

What does the model learn? It seems to be using the scene category label.

What does the model learn?

Model Learns Scene Label In a baseball game. A ball is most likely object to be hit. In a beach. Likely to be surfing.

What does the model learn?

What does the model learn?

Model Learns Co-Occurrence of objects What objects are likely to be on top of a car.

What does the model learn?

What does the model learn?

Model learns actions associated with objects A dog is likely to be eating. A pizza is likely to be eaten by a man.

Conclusion The model is learning many shortcut ways of answering questions. It might be benefited by getting a list of objects and the scene category of the image, this can be tested using ground truth annotations.