Week 3 Presentation Ngoc Ta Aidean Sharghi

Research Papers Bidirectional Attentive Fusion With Context Gating for Dense Video Captioning End-to-End Dense Video Captioning with Masked Transformer Video Captioning via Hierarchical Reinforcement Learning M3 : Multimodal Memory Modelling for Video Captioning

End-to-End Dense Video Captioning with Masked Transformer Code available: https://github.com/salesforce/densecap Dataset: Activitynet Captions and YouCookII Method: Using an end-to-end model which is composed of an encoder and two decoders. Encoder encodes Proposal decoder Captioning decoder Using self-attention for dense video captioning 1. Encoder encodes the input video to proper visual representations. 2. Proposal decoder then decodes from this representation with different anchors to form video event proposals. 3. Captioning decoder employs a differentiable masking network to restrict its attention to the proposal event, ensures the consistency between the proposal and captioning during training. Using self-attention (enables the use of efficient non-recurrent structure during encoding)

Video Captioning via Hierarchical Reinforcement Learning Code: N/A Dataset: MSR-VTT dataset and Charades dataset Method: Aims at improving the fine-grained generation of video descriptions with rich activities. Gains better improvement on detailed descriptions of longer videos. Pretrained CNN model Low-level Bi-LSTM2 encoder High-level LSTM3 encoder HRL works as a decoder which is composed of three components: a low-level worker, a high-level manager, and an internal critic The whole pipeline terminates once an end of sentence token is reached. Video frame features are first extracted by a pretrained CNN model Passed through a low-level Bi-LSTM2 encoder and a high-level LSTM3 encoder to obtain low-level and high-level encoder output. HRL works as a decoder which is composed of three components: a low-level worker, a high-level manager, and an internal critic The whole pipeline terminates once an end of sentence token is reached.

M3 : Multimodal Memory Modelling for Video Captioning Code: N/A Dataset: MSVD and MSR-VTT Method: Using M3 which contains: CNN-based video encoder Multimodal memory LSTM-based text decoder CNN-based video encoder first extracts video frame/clip features using pretrained 2D/3D CNNs LSTM-based text decoder models the sentence generation then writes the updated representation to the memory. Multimodal memory contains a memory matrix Mem to interact with video and sentence Procedure: 1. Write hidden states to update memory 2. Read the updated memory content to perform soft-attention 3. Write selected visual information to update memory again 4. Read the updated memory content for next word prediction

Bidirectional Attentive Fusion With Context Gating for Dense Video Captioning Code available: https://github.com/JaywongWang/DenseVideoCaptioning Dataset : ActivityNet Captions (20k YouTube untrimmed videos from real life, 120 seconds long on average.) Goals: to automatically localize events in video and describe each one with sentence Method : Using a novel bidirectional proposal framework( Bidirectional SST) to encode both past and future contexts, with the motivation that both past and future contexts help better localize the current event.

Bidirectional Attentive Fusion With Context Gating for Dense Video Captioning In the forward pass, they learn k binary classifiers corresponding to k anchors densely at each time step In backward pass we reverse both video sequence input and predict proposals backward. This means that the forward pass encodes past context and current event information, while the backward pass encodes future context and current event information Finally they merge proposal scores for the same predictions from the two passes and output final proposals.

Bidirectional Attentive Fusion With Context Gating for Dense Video Captioning

Bidirectional Attentive Fusion With Context Gating for Dense Video Captioning To construct more discriminative proposal representation, they fused proposal state information and detected video content together which help discriminate highly overlapped events. To output more confident results, they used joint ranking technique to select high-confidence proposal caption pairs by taking both proposal score and caption confidence into consideration.

Bidirectional Attentive Fusion With Context Gating for Dense Video Captioning

THANK YOU