1. Social Multimedia Analytics A Case Study on Microblog Sentiment Prediction
Rongrong Ji
Director, Intelligent Multimedia Laboratory
Dean Assistant, School of Information Science and Technology

2. Outline Introduction Visual Sentiment Ontology Background
Research Interests
Visual Sentiment Ontology
Motivation
Design Principle
Demo

3. Outline Introduction Visual Sentiment Ontology Background
Research Interests
Visual Sentiment Ontology
Motivation
Design Principle
Demo

4. Background 1. Research Interest and Collaborators
My work is about…
Visual Big Data
Mobile Visual Search
Social Multimedia Analysis
Scene Understanding (3D/Depth/Saliency…)
Collaborators
Professor Shih-Fu Chang and Steven Feiner (Columbia University)
Professor Wen Gao (Peking University) and Hongxun Yao (Harbin Institute of Technology)
Dr. Xing Xie and Dr. Yong Rui (Microsoft Research)
Professor Qi Tian (University of Texas at San Antonio)

5. Outline Introduction Visual Sentiment Ontology Background
Research Interests
Visual Sentiment Ontology
Motivation
Design Principle
Demo

6. Research Interest 1. Mobile Visual Search
Mobile Visual Search and Compact Descriptor
Scenarios
Issue
Query delivery latency in mobile visual search
Solution
Directly exact and send compact visual descriptor from the mobile end
Learning to compress the original descriptor based on the mobile context
Pub.
IJCV, TIP, TMM, CVPR 12, ACM MM 11, and IJCA 11

7. Research Interest 1. Mobile Visual Search
Interactive Query Formulation
Active Query Sensing in Mobile Visual Search
Scenario
Mobile Location Recognition
Point
How the offline scene analysis can help to guide online user to take as few query as possible to find the target location
Pub.
ACM Multimedia 2011 Best Paper
ACM Trans. Multimedia Computing

8. Research Interest 2. Massive Scale Visual Search
My interests about “Visual Big Data”
Large-Scale Visual Search and Recognition
Hierarchical Vector Space Quantization Error Compensation for location search
Quantization Tree Transfer Learning across datasets
CVPR 2009, IEEE Multimedia 2011
Embedding Semantics into Visual Feature Space Quantization
CVPR 2010 (Oral), TIP
Supervised Hashing with Kernels
CVPR 2012 (Oral)

9. Research Interest 3. Social Multimedia Analysis
Landmark Mining from Blogs and Flickr
HITS based canonical view selection
Sparse Representation based canonical view selection
Twitter/Weibo Sentiment Analysis
Pub.
ACM Multimedia 2013 (Brave New Idea Track), ACM Multimedia 2009 (Oral)

10. Research Interest 4. Scene Understanding
Single Image Depth Estimation
Discriminative Scene Parsing and Depth Estimation
Joint Depth and Semantic Parsing with Structure SVM
Multi-User Semantic-Aware Mobile Augmented Reality
Visual Saliency
Pub.
CVPR 2013, CVPR 2012 (Oral)

11. Outline Introduction Visual Sentiment Ontology Background
Research Interests
Visual Sentiment Ontology
Motivation
Design Principle
Demo

12. Recent Work 1. Visual Sentiment Ontology
Motivation
Massive and Ever Increasing Social MultiMedia
Images (300 million photos uploaded to Facebook every day)
Videos (4 billion videos watched per month on YouTube)

13. Recent Work 1. Visual Sentiment Ontology
Motivation
One of the most important purpose of social media is to express the user opinion

14. Recent Work 1. Visual Sentiment Ontology
Motivation
But…
Text based emotion/sentiment/opinion analysis tool is troublesome when facing microblogs
@BarackObama: Four more years.
@Brynn4NY: Rollercoaster at sea.

15. Recent Work 1. Visual Sentiment Ontology
Motivation
So…
Can we reliably detect sentiments and/or emotions within images?
Web + big data + computer vision + psychology
We share a Visual Sentiment Ontology to the community
1200-dim SentiBank Detector
Ontology Dataset
Code
A picture is worth one thousand words, but what words should be used to describe the sentiments and emotions conveyed in the increasingly popular social multimedia?

16. Recent Work 1. Visual Sentiment Ontology
Design Principle
Each concept should
have a strong correlation with sentiment reflected in the image
be interpretable by human and understandable by machine
The outputs of all concepts should
ensure a good coverage of potential emotions and concepts in images
a middle-level representation to predict the sentiment of an image
Question:
What is the sentiment(opinion) that the users want to express through photo uploading
The Idea:
Design a concept dictionary to discover the positive/negative sentiments (if any) of images

17. Recent Work 1. Visual Sentiment Ontology
Initialization: Plutchik's Wheel of Emotion model

18. Recent Work 1. Visual Sentiment Ontology
Step 1
data mining to discover visual sentiments in social media
Crawl initial words by Plutchik’s “Wheel of Emotion” categories from YouTube and Flickr
Initial concept dictionary
Crowdsourced Websites
Emotion Keyword Queries

19. Recent Work 1. Visual Sentiment Ontology
Step 1
data mining to discover visual sentiments in social media
Identify frequent photo tags related to emotions
Detectable nouns are not emotion-related!
Emotion-related adjectives are not detectable!
So which 1000 concepts to focus in pictures?

20. Recent Work 1. Visual Sentiment Ontology
Step 1
data mining to discover visual sentiments in social media
Adjective-Noun Pair
Adjective (268): needed for expressing emotions
frequent positive Adj: beautiful, amazing, cute
frequent negative Adj: sad, angry, dark
Nouns (1187): feasible for computer vision
Noun categories: people, places, animals, food, objects, weather
Standard steps
remove named entities like “hot dog” via wikipedia
Choose sentiment rich ANP concepts by tools “Senti‐WordNet” “SentiStrength”

21. Recent Work 1. Visual Sentiment Ontology
Step 1
data mining to discover visual sentiments in social media
Adjective-Noun Pair

22. Recent Work 1. Visual Sentiment Ontology
Browser:

23. Recent Work 1. Visual Sentiment Ontology
Browser:

24. Recent Work 1. Visual Sentiment Ontology
Step 2
Concept Detector Training and Filtering
Visual Detectors
Performance
Filtering
Final Concept Dictionary

25. Recent Work 1. Visual Sentiment Ontology
Step 2
Concept Detector Training and Filtering
LibSVM, 5‐fold cross validation
Features
RGB Color Histogram (3x256 dim.)
GIST descriptor (512 dim.)
Local Binary Pattern (52 dim.)
SIFT Bag‐of‐Words (1,000 codewords, 2‐layer spatial pyramid, max pooling)
Classemes descriptor (2,659 dim.)

26. Recent Work 1. Visual Sentiment Ontology
Step 2
Concept Detector Training and Filtering
Good Results Not Good Results

27. Recent Work 1. Visual Sentiment Ontology
Application
Live Twitter Stream Sentiment Prediction
True stuff. I have mad respect for all the
ladies that DO NOT give in to abortion
Ouch mr police man
#groundzero #hurricanesandy
#newjersey

28. Recent Work 1. Visual Sentiment Ontology
Application
Live Twitter Stream Sentiment Prediction
2000 tweets with images
Two-way (positive/negative) prediction

29. Recent Work 1. Visual Sentiment Ontology
Demo

30. Thank You!
imt.xmu.edu.cn/