1. CRCV REU 2019
Aaron Honculada

2. Model

3. Model

4. Model

5. Quantitative Results Model Feature Type Q Q + V LSTM ------- 42.48%

6. Quantitative Results Model Feature Type Q Q + V LSTM ------- 42.48%
VCPT
42.67%

7. Quantitative Results Model Feature Type Q Q + V LSTM ------- 42.48%
VCPT
42.67%
CNN + LSTM
FC (ResNet101)
42.73%

8. Quantitative Results Model Feature Type Q Q + V LSTM ------- 42.48%
VCPT
42.67%
CNN + LSTM
FC (ResNet101)
42.73%
FC (I3D)
42.86%

9. Quantitative Results Model Feature Type Q Q + V LSTM ------- 42.48%
VCPT
42.67%
CNN + LSTM
FC (ResNet101)
42.73%
FC (I3D)
42.86%
Conv (I3D)
42.80%

10. Qualitative Results Question Correct Answer Model: BiLSTM
Model: CNN + BiLSTM
What did Bernadette say to Howard after she told him that the next person to kick him will be her ?
Bernadette told Howard , Good night .
At the Vet .
Bernadett told Leonard , Good night
What did Caldwell confess to after he saw Hayley shoot Castle ?
Caldwell confessed to knowing the cop
She changed her seat
Caldwell confessed to killing the cop

11. Qualitative Results Question Correct Answer Model: BiLSTM
Model: CNN + BiLSTM
What did Bernadette say to Howard after she told him that the next person to kick him will be her ?
Bernadette told Howard , Good night .
At the Vet .
Bernadett told Leonard , Good night
What did Caldwell confess to after he saw Hayley shoot Castle ?
Caldwell confessed to knowing the cop
She changed her seat
Caldwell confessed to killing the cop
What injury was Taub tending to when he was treating Adams ?
A cut on her head
Decent meal , hot shower , and a real bed
A cut on her arm
What did Cameron say can cause malnutrition when Cameron was talking to House and her other colleagues ?
excruciating stomach pain
MVP
sick feeling in stomach

12. Qualitative Results Question Correct Answer Model: BiLSTM
Model: CNN + BiLSTM
What did Bernadette say to Howard after she told him that the next person to kick him will be her ?
Bernadette told Howard , Good night .
At the Vet .
Bernadett told Leonard , Good night
What did Caldwell confess to after he saw Hayley shoot Castle ?
Caldwell confessed to knowing the cop
She changed her seat
Caldwell confessed to killing the cop
What injury was Taub tending to when he was treating Adams ?
A cut on her head
Decent meal , hot shower , and a real bed
A cut on her arm
What did Cameron say can cause malnutrition when Cameron was talking to House and her other colleagues ?
excruciating stomach pain
MVP
sick feeling in stomach
What causes Castle to have flashbacks when eating ?
Myjapchae
Salad
What news does Frank give the breakroom after he runs throught the hallway ?
Frank is now the father of an baby boy .
Frank has is now the father of an baby girl .
Frank has yelled , Go Redskins and fell collapsed .

13. Qualitative Results Question Correct Answer Model: BiLSTM
Model: CNN + BiLSTM
What did Bernadette say to Howard after she told him that the next person to kick him will be her ?
Bernadette told Howard , Good night .
At the Vet .
Bernadett told Leonard , Good night
What did Caldwell confess to after he saw Hayley shoot Castle ?
Caldwell confessed to knowing the cop
She changed her seat
Caldwell confessed to killing the cop
What injury was Taub tending to when he was treating Adams ?
A cut on her head
Decent meal , hot shower , and a real bed
A cut on her arm
What did Cameron say can cause malnutrition when Cameron was talking to House and her other colleagues ?
excruciating stomach pain
MVP
sick feeling in stomach
What causes Castle to have flashbacks when eating ?
Myjapchae
Salad
What news does Frank give the breakroom after he runs throught the hallway ?
Frank is now the father of an baby boy .
Frank has is now the father of an baby girl .
Frank has yelled , Go Redskins and fell collapsed .
What color is Bernadette blouse when she was driving ?
Green
Gray
Blue
What did Rachel do before she started talking to Ross ?
Rachel pull Ross to sit on a table next to her
Rachel pull Ross to sit on a couch next to her

14. Next Steps
Parse the Questions to extract subqueries and essential information
Retrieve information from video features for each subquery
Train with Multi-Margin Loss Function
Merge