Textual Video Prediction Week 2 REU Student: Emily Cosgrove Graduate Student: Amir Mazaheri Professor: Dr. Shah

Project Overview Video Prediction Generative Adversarial Networks (GANs) Textual information for video prediction Goal: Enhanced video prediction

Progress Large Scale Movie Description Challenge (LSMDC) challenge dataset 128,000 videos Each 2-20 seconds Comes with text Clipped each video into 1 second clips (about 30 frames) Now have 359,000 video clips Text coming with each We use standard training-validation-test split (used in LSMDC- 2016) 90% data as training, 5% validation, and 5% testing

Annotation: trying to lighten the mood. Someone smiles sheepishly

1 2 3 4 5 6 LSTM LSTM LSTM LSTM LSTM LSTM CNN CNN CNN CNN CNN CNN CNN DE-CONV DE-CONV DE-CONV LSTM LSTM LSTM LSTM LSTM LSTM CNN CNN CNN CNN CNN CNN CNN CNN CNN CNN CNN 1 2 3 4 5 6

Progress Two Research Papers Generative Adversarial Text to Image Synthesis (Scott Reed, et.) Uses GANs Developed a model to generate images based on textual description Convolutional LSTM Network: A Machine Learning Approach to Precipitation Nowcasting (Xingjian Shi, et.) Precipitation Nowcasting: challenging weather forecasting problem Proposed convolutional LSTM (ConvLSTM)

Next Week Make model more complete by adding Adversarial loss and text