NOTE: To change the image on this slide, select the picture and delete it. Then click the Pictures icon in the placeholder to insert your own image. SHOW AND TELL: A NEURAL IMAGE CAPTION GENERATOR Course Project – Computer Vision and Image Processing (CS676A) Kriti Joshi & Pramod Chunduri

Image Captioning  Generate natural sentences to describe an image "man in black shirt is playing guitar.""little girl is eating piece of cake."

Tasks involved  Detection of objects  Correlation between objects  Attributes of these objects  Activities these objects are involved in  Formation of sentences from above knowledge "man in black shirt is playing guitar."

Approach Encoder RNNDecoder RNN Source Language Target Language Fixed length vector Deep CNN Language generating RNN Fixed length vector "man in black shirt is playing guitar." Neural Image Caption (NIC)

Recurrent Neural Networks (RNN) Pros:  Sequential Data  Variable caption size Cons:  Vanishing gradient problem Solution:  LSTM  Forget gates

Long Short Term Memory (LSTM)

Sampling Extensions  Beam search (choosing parameter k)  Input image in each iteration of LSTM along

Thank You Questions ??