1. Learn to Comment Mentor: Mahdi M. Kalayeh
REU Students: Lance Lebanoff | David Hill | Jonathan Pham

2. Problem Definition
Given an image, generate comments about the image that mimic human comments.
Use a combination of existing computer vision techniques to extract the features from the image:
Object detection
Scene understanding
Sentiment analysis
Train a deep neural network.

3. Useful Ideas From Previous Works
Deep Learning structures
CNN, LSTM
Long Short Term Memory networks (LSTM)
Accepts temporal sequences of arbitrary length
Output natural language word by word
Natural language processing for sentence descriptions
Sentiment analysis from Sentibank

4. LRCNN
Donahue, Jeff
Long-term Recurrent Convolutional Networks for Visual Recognition and Description

5. LSTM
Donahue, Jeff
Long-term Recurrent Convolutional Networks for Visual Recognition and Description

6. Is deep learning enough?
Is an end-to-end neural network sufficient?
Training from scratch
Issue of overfitting
Pre-trained

7. Descriptions vs. Comments
Image-sentence fragment alignment
Comments can refer to concepts beyond the context of the image.
Pop culture, current events, visual aesthetic, ...
Unlike descriptions, comments confer sentiment about their subjects.

8. Extracting sentiment from an image (SentiBank).

9. Anticipated Challenges
Generating comments on an image is a harder task than describing its visual content.
Many possible ‘divergent’ comments.
Standard NLP metrics like BLEU will not work.
Data Collection
Comments on images are often replies to other comments.
Captions on pictures often influence the nature of the comments.
We will most likely need to clean our own data sets by removing irrelevant comments.

10. Division of Tasks Lance: Sentiment Analysis from textual content
Studying possible data collection approaches
Jonathan:
Sentiment Analysis from visual content
David:
Deep Learning (Caffe, LRCNN)