1. Textual Video Prediction Week 2
REU Student: Emily Cosgrove
Graduate Student: Amir Mazaheri
Professor: Dr. Shah

2. Project Overview Video Prediction
Generative Adversarial Networks (GANs)
Textual information for video prediction
Goal: Enhanced video prediction

3. Progress
Large Scale Movie Description Challenge (LSMDC) challenge dataset
128,000 videos
Each 2-20 seconds
Comes with text
Clipped each video into 1 second clips (about 30 frames)
Now have 359,000 video clips
Text coming with each
We use standard training-validation-test split (used in LSMDC )
90% data as training, 5% validation, and 5% testing

4. Annotation:
trying to lighten the mood. Someone smiles sheepishly

5. 1 2 3 4 5 6 LSTM LSTM LSTM LSTM LSTM LSTM CNN CNN CNN CNN CNN CNN CNN
DE-CONV
DE-CONV
DE-CONV
LSTM
LSTM
LSTM
LSTM
LSTM
LSTM
CNN
CNN
CNN
CNN
CNN
CNN
CNN
CNN
CNN
CNN
CNN 1 2 3 4 5 6

6. Progress Two Research Papers
Generative Adversarial Text to Image Synthesis (Scott Reed, et.)
Uses GANs
Developed a model to generate images based on textual description
Convolutional LSTM Network: A Machine Learning Approach to Precipitation Nowcasting (Xingjian Shi, et.)
Precipitation Nowcasting: challenging weather forecasting problem
Proposed convolutional LSTM (ConvLSTM)

7. Next Week
Make model more complete by adding Adversarial loss and text