CRCV REU 2019 Week 8 Aaron Honculada

Current Approach Parsing Question into Subqueries How to represent video to utilize the information

Literature Review Neural-Symbolic Concept Learner Visual Perception Module Mask R-CNN The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision (Tenenbaum, ICLR 2019)

Neural-Symbolic Concept Learner Semantic Parsing Question to program Hierarchy of operations concepts as parameters The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision (Tenenbaum, ICLR 2019)

Neural-Symbolic Concept Learner Domain Specific Language Requires unique referral object The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision (Tenenbaum, ICLR 2019)

Dependency Parsing Sentence to Dependency Tree Dependency from Grammatical Relation Possible Multi-task model VQA Generate Dependency Tree

Dependency Parsing Example What could happen to the newborn , Emily , after Izzie administers the ECMO ?

Dependency Parsing Locate keywords in questions With keywords formulate subqueries Biaffine Classifier and BiLSTM Deep Biaffine Attention for Neural Dependency Parsing (Dozat, ICLR 2017)

Video Representation Cannot fully answer from visual concepts alone Contextualize objects in scene Color Belongs to person or scene Places365 Scene Attributes Indoor/ Outdoor