CRCV REU 2019 Aaron Honculada

Model

Model

Model

Quantitative Results Model Feature Type Q Q + V LSTM ------- 42.48% ---------

Quantitative Results Model Feature Type Q Q + V LSTM ------- 42.48% --------- VCPT 42.67%

Quantitative Results Model Feature Type Q Q + V LSTM ------- 42.48% --------- VCPT 42.67% CNN + LSTM FC (ResNet101) 42.73%

Quantitative Results Model Feature Type Q Q + V LSTM ------- 42.48% --------- VCPT 42.67% CNN + LSTM FC (ResNet101) 42.73% FC (I3D) 42.86%

Quantitative Results Model Feature Type Q Q + V LSTM ------- 42.48% --------- VCPT 42.67% CNN + LSTM FC (ResNet101) 42.73% FC (I3D) 42.86% Conv (I3D) 42.80%

Qualitative Results Question Correct Answer Model: BiLSTM Model: CNN + BiLSTM What did Bernadette say to Howard after she told him that the next person to kick him will be her ? Bernadette told Howard , Good night . At the Vet . Bernadett told Leonard , Good night What did Caldwell confess to after he saw Hayley shoot Castle ? Caldwell confessed to knowing the cop She changed her seat Caldwell confessed to killing the cop

Qualitative Results Question Correct Answer Model: BiLSTM Model: CNN + BiLSTM What did Bernadette say to Howard after she told him that the next person to kick him will be her ? Bernadette told Howard , Good night . At the Vet . Bernadett told Leonard , Good night What did Caldwell confess to after he saw Hayley shoot Castle ? Caldwell confessed to knowing the cop She changed her seat Caldwell confessed to killing the cop What injury was Taub tending to when he was treating Adams ? A cut on her head Decent meal , hot shower , and a real bed A cut on her arm What did Cameron say can cause malnutrition when Cameron was talking to House and her other colleagues ? excruciating stomach pain MVP sick feeling in stomach

Qualitative Results Question Correct Answer Model: BiLSTM Model: CNN + BiLSTM What did Bernadette say to Howard after she told him that the next person to kick him will be her ? Bernadette told Howard , Good night . At the Vet . Bernadett told Leonard , Good night What did Caldwell confess to after he saw Hayley shoot Castle ? Caldwell confessed to knowing the cop She changed her seat Caldwell confessed to killing the cop What injury was Taub tending to when he was treating Adams ? A cut on her head Decent meal , hot shower , and a real bed A cut on her arm What did Cameron say can cause malnutrition when Cameron was talking to House and her other colleagues ? excruciating stomach pain MVP sick feeling in stomach What causes Castle to have flashbacks when eating ? Myjapchae Salad What news does Frank give the breakroom after he runs throught the hallway ? Frank is now the father of an baby boy . Frank has is now the father of an baby girl . Frank has yelled , Go Redskins and fell collapsed .

Qualitative Results Question Correct Answer Model: BiLSTM Model: CNN + BiLSTM What did Bernadette say to Howard after she told him that the next person to kick him will be her ? Bernadette told Howard , Good night . At the Vet . Bernadett told Leonard , Good night What did Caldwell confess to after he saw Hayley shoot Castle ? Caldwell confessed to knowing the cop She changed her seat Caldwell confessed to killing the cop What injury was Taub tending to when he was treating Adams ? A cut on her head Decent meal , hot shower , and a real bed A cut on her arm What did Cameron say can cause malnutrition when Cameron was talking to House and her other colleagues ? excruciating stomach pain MVP sick feeling in stomach What causes Castle to have flashbacks when eating ? Myjapchae Salad What news does Frank give the breakroom after he runs throught the hallway ? Frank is now the father of an baby boy . Frank has is now the father of an baby girl . Frank has yelled , Go Redskins and fell collapsed . What color is Bernadette blouse when she was driving ? Green Gray Blue What did Rachel do before she started talking to Ross ? Rachel pull Ross to sit on a table next to her Rachel pull Ross to sit on a couch next to her

Next Steps Parse the Questions to extract subqueries and essential information Retrieve information from video features for each subquery Train with Multi-Margin Loss Function Merge