1. Sequence to Sequence Video to Text
Subhashini Venugopalan, Marcus Rohrbach, Jeff Donahue, Raymond Mooney, Trevor Darrell, Kate Saenko
Presented by Dewal Gupta
UCSD CSE 291G, Winter 2019

2. BACKGROUND Challenge: Create a description for a given video
Important in:
describing videos for blind
human-robot interactions
Challenging because:
diverse set of scenes, actions
necessary to recognize salient action in context
images don’t have to face the

3. PREVIOUS WORK: Template Models
Tag video with captions and use as bag of words
Two stage pipeline:
first: tag video with semantic information on objects, actions
treated as a classification problem
FGM labels subject, verb, object, place
second: generate sentence from semantic information
S2VT approach: avoids separating content identification from sentence generation
Integrating Language and Vision: to Generate Natural Language Descriptions of Videos in the Wild - Mooney et al., 2014

4. PREVIOUS WORK: Mean Pooling
CNN trained on object classification (subset of ImageNet)
2 layer LSTM with video and previous word as input
Ignores video frame ordering
Translating Videos to Natural Language Using Deep Recurrent Neural Networks Mooney et al., 2015

5. PREVIOUS WORK: Exploiting Temporal Structure
Encoder:
train 3D ConvNet on action recognition
fixed frame input
exploits local temporal structure
Describing Videos by Exploiting Temporal Structure Videos in the Wild
Courville et al., 2015

6. PREVIOUS WORK: Exploiting Temporal Structure
Decoder:
Similar to our HW 2
Exploits global temporal structure
Describing Videos by Exploiting Temporal Structure Videos in the Wild
Courville et al., 2015

7. GOAL End to End differentiable model that can:
Handle variable video length (i.e. variable input length)
Learn temporal structure
Learn a language model that is capable of generating descriptive sentences

8. MODEL: LSTM Single LSTM network 2 layer LSTM network
1000 hidden units (ht)
red layer: models visual elements
green layer: models linguistic elements
ILSVRC-2012 object classification subset of the ImageNet dataset [30]

9. MODEL: VGG-16

10. MODEL: AlexNet
Used for RGB & Flow!

11. MODEL: Details Use Text Embedding (of 500 dimensions)
self-trained, simple linear transformation
RGB networks are pre-trained on subset of ImageNet data
Used networks from the original works
Optical Flow pretrained on UCF101 dataset
Action Classification Task
Original work from ‘Action Tubes’
All layers are frozen except last layers for training
Flow and RGB combined by “shallow fusion technique”
alpha is tuned on the validation set

12. DATASETS 3 datasets used: Microsoft Video Description corpus (MSVD)
MPII Movie Description Corpus (MPII-MD)
Montreal Video Annotation Dataset (M-VAD)
MSVD: web clips with human annotations
MPII-MD: Hollywood clips with descriptions from script & audio (originally for the visually impaired)
M-VAD: Hollywood clips with audio descriptions
All three have single sentence descriptions

13. DATASETS: Metrics Authors use METEOR metric
uses exact token, stemmed token and WordNet synonym matches
better correlation with human judgement than BLEU or ROUGE
out performs CIDEr when fewer references
datasets only had 1 reference
where:
m is unigram (or n-gram) matches after alignment
wr is length of reference
wt is length of candidate
BLEU does not take recall into consideration directly - rather just penalizes brevity

14. RESULTS: MSVD FGM is template based not very descriptive
predicts a noun, verb, object, place
builds sentence off template

15. RESULTS: MSVD
Mean Pool based method
very similar to author’s method

16. RESULTS: MSVD Temporal Attention method
Encoder/Decoder using attention

17. RESULTS: Frame ordering
Training with random ordering of frames results in “considerably lower” performance

18. RESULTS: Optical Flow
Flow results in better performance only when combined with RGB (& not when used alone)
Flow can be very different even for same activities
Flow can’t account for polysemous words like “play” - eg. “play guitar” vs “play golf”
person vs panda eating

19. RESULTS: SOTA
Authors claim accurate comparison is with GoogleNet with NO 3D-CNN (global temporal attention)
questionable claim
person vs panda eating

20. Results: MPII-MD, M-VAD
Similar performance to Visual-Labels
VL uses more semantic information (eg. object detection) but no temporal information

21. Results: Edit Distance
Levenshtein Distance: represents edit distance between two strings
42.9% of generated samples match exactly with a sentence in the training corpus of MSVD
model struggles to learn MVAD

22. CRITICISM Model fails to learn temporal relations
performs nearly as well as mean pooling technique that makes no use of temporal relations
Model struggles on MVAD dataset for some reason more than other
Authors should have used BLEU and/or CIDEr scores as well (other studies have them)
Conduct user study (where human looks at captions)?
Could improve by using better text embeddings?

23. FURTHER WORK Use Inception ResNet v2 as backbone CNN
Train CNN against mined video “attributes”
Achieve +5% METEOR score on MSVD
Same architecture
End-to-End Video Captioning with Multitask Reinforcement Learning - Li & Gong, 2019

24. FURTHER WORK Use 3D CNN to get better clip embeddings instead of LSTMs
proven better in activity recognition
Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification - Xie et al., 2017

25. CONCLUSION Authors build an end to end differentiable model that can:
Handle variable video length (i.e. variable input length)
Learn temporal structure
Learn a language model that is capable of generating descriptive sentences
Has become a baseline for many video captioning works

26. EXAMPLES