Visual Question Answering Aaron Honculada, Aisha Urooj Khan, Dr Visual Question Answering Aaron Honculada, Aisha Urooj Khan, Dr. Niels Lobo, Dr. Mubarak Shah Results Background Architectures Visual Question Answer (VQA) is at the intersection of Computer Vision and Natural Language Processing. Given an image or video and questions, VQA aims to design systems capable of answering questions about or related to the visual information. Baseline Model TVQA Prediction A1) Soft drink Our Prediction A4) Beer What was Raj drinking when he was sitting at the table with Leonard and Howard ? A0) Lemonade A2) Water A3) Milk Dataset TVQA Prediction A1) Chandler was eating a cheeseburger . Our Prediction A4) Chandler was smoking a cigarette. What was Chandler doing when Phoebe turned around on her stool ? A0) Chandler was looking at stars . A2) Chandler was making a cake. A3) Chandler was doing jumping jacks. TVQA Dataset: 6 TV Series 460 hours of video 21,793 clips 152,545 Question and Answer Pairs 7 Questions per clip 5 Multiple choice answers 5 Question Families Included Subtitles Video Features Visual Concepts Attention Based Model TVQA Prediction A0) 300 Our Prediction What number did House write on the board after Taub suggested a MRA ? A1) 10 A2) 1 A3) 60 A4) 12 Results Model Feature Accuracy LSTM N/A 42.48% VCPT 42.67% CNN+LSTM ResNet (FC) 42.73% I3D (FC) 42.86% I3D (Conv) 42.59% Attention TVQA 43.78% TVQA Prediction A3) Purple Our Prediction A2) red What color is the shirt Castle is wearing when he says, “Ashley , I'm sorry.”? A0) black A1) green A4) yellow Acknowledgement We would like to thank the National Science Foundation for funding this project. NSF IIS-1757858