Social Multimedia Analytics A Case Study on Microblog Sentiment Prediction Rongrong Ji Director, Intelligent Multimedia Laboratory Dean Assistant, School of Information Science and Technology

Outline Introduction Visual Sentiment Ontology Background Research Interests Visual Sentiment Ontology Motivation Design Principle Demo

Outline Introduction Visual Sentiment Ontology Background Research Interests Visual Sentiment Ontology Motivation Design Principle Demo

Background 1. Research Interest and Collaborators My work is about… Visual Big Data Mobile Visual Search Social Multimedia Analysis Scene Understanding (3D/Depth/Saliency…) Collaborators Professor Shih-Fu Chang and Steven Feiner (Columbia University) Professor Wen Gao (Peking University) and Hongxun Yao (Harbin Institute of Technology) Dr. Xing Xie and Dr. Yong Rui (Microsoft Research) Professor Qi Tian (University of Texas at San Antonio)

Outline Introduction Visual Sentiment Ontology Background Research Interests Visual Sentiment Ontology Motivation Design Principle Demo

Research Interest 1. Mobile Visual Search Mobile Visual Search and Compact Descriptor Scenarios Issue Query delivery latency in mobile visual search Solution Directly exact and send compact visual descriptor from the mobile end Learning to compress the original descriptor based on the mobile context Pub. IJCV, TIP, TMM, CVPR 12, ACM MM 11, and IJCA 11

Research Interest 1. Mobile Visual Search Interactive Query Formulation Active Query Sensing in Mobile Visual Search Scenario Mobile Location Recognition Point How the offline scene analysis can help to guide online user to take as few query as possible to find the target location Pub. ACM Multimedia 2011 Best Paper ACM Trans. Multimedia Computing

Research Interest 2. Massive Scale Visual Search My interests about “Visual Big Data” Large-Scale Visual Search and Recognition Hierarchical Vector Space Quantization Error Compensation for location search Quantization Tree Transfer Learning across datasets CVPR 2009, IEEE Multimedia 2011 Embedding Semantics into Visual Feature Space Quantization CVPR 2010 (Oral), TIP Supervised Hashing with Kernels CVPR 2012 (Oral)

Research Interest 3. Social Multimedia Analysis Landmark Mining from Blogs and Flickr HITS based canonical view selection Sparse Representation based canonical view selection Twitter/Weibo Sentiment Analysis Pub. ACM Multimedia 2013 (Brave New Idea Track), ACM Multimedia 2009 (Oral)

Research Interest 4. Scene Understanding Single Image Depth Estimation Discriminative Scene Parsing and Depth Estimation Joint Depth and Semantic Parsing with Structure SVM Multi-User Semantic-Aware Mobile Augmented Reality Visual Saliency Pub. CVPR 2013, CVPR 2012 (Oral)

Outline Introduction Visual Sentiment Ontology Background Research Interests Visual Sentiment Ontology Motivation Design Principle Demo

Recent Work 1. Visual Sentiment Ontology Motivation Massive and Ever Increasing Social MultiMedia Images (300 million photos uploaded to Facebook every day) Videos (4 billion videos watched per month on YouTube)

Recent Work 1. Visual Sentiment Ontology Motivation One of the most important purpose of social media is to express the user opinion

Recent Work 1. Visual Sentiment Ontology Motivation But… Text based emotion/sentiment/opinion analysis tool is troublesome when facing microblogs @BarackObama: Four more years. @Brynn4NY: Rollercoaster at sea.

Recent Work 1. Visual Sentiment Ontology Motivation So… Can we reliably detect sentiments and/or emotions within images? Web + big data + computer vision + psychology We share a Visual Sentiment Ontology to the community 1200-dim SentiBank Detector Ontology Dataset Code A picture is worth one thousand words, but what words should be used to describe the sentiments and emotions conveyed in the increasingly popular social multimedia? http://visual-sentiment-ontology.appspot.com

Recent Work 1. Visual Sentiment Ontology Design Principle Each concept should have a strong correlation with sentiment reflected in the image be interpretable by human and understandable by machine The outputs of all concepts should ensure a good coverage of potential emotions and concepts in images a middle-level representation to predict the sentiment of an image Question: What is the sentiment(opinion) that the users want to express through photo uploading The Idea: Design a concept dictionary to discover the positive/negative sentiments (if any) of images

Recent Work 1. Visual Sentiment Ontology Initialization: Plutchik's Wheel of Emotion model

Recent Work 1. Visual Sentiment Ontology Step 1 data mining to discover visual sentiments in social media Crawl initial words by Plutchik’s “Wheel of Emotion” categories from YouTube and Flickr Initial concept dictionary Crowdsourced Websites Emotion Keyword Queries

Recent Work 1. Visual Sentiment Ontology Step 1 data mining to discover visual sentiments in social media Identify frequent photo tags related to emotions Detectable nouns are not emotion-related! Emotion-related adjectives are not detectable! So which 1000 concepts to focus in pictures?

Recent Work 1. Visual Sentiment Ontology Step 1 data mining to discover visual sentiments in social media Adjective-Noun Pair Adjective (268): needed for expressing emotions frequent positive Adj: beautiful, amazing, cute frequent negative Adj: sad, angry, dark Nouns (1187): feasible for computer vision Noun categories: people, places, animals, food, objects, weather Standard steps remove named entities like “hot dog” via wikipedia Choose sentiment rich ANP concepts by tools “Senti‐WordNet” “SentiStrength”

Recent Work 1. Visual Sentiment Ontology Step 1 data mining to discover visual sentiments in social media Adjective-Noun Pair

Recent Work 1. Visual Sentiment Ontology Browser: http://visual-sentiment-ontology.appspot.com

Recent Work 1. Visual Sentiment Ontology Browser: http://visual-sentiment-ontology.appspot.com

Recent Work 1. Visual Sentiment Ontology Step 2 Concept Detector Training and Filtering Visual Detectors Performance Filtering Final Concept Dictionary

Recent Work 1. Visual Sentiment Ontology Step 2 Concept Detector Training and Filtering LibSVM, 5‐fold cross validation Features RGB Color Histogram (3x256 dim.) GIST descriptor (512 dim.) Local Binary Pattern (52 dim.) SIFT Bag‐of‐Words (1,000 codewords, 2‐layer spatial pyramid, max pooling) Classemes descriptor (2,659 dim.)

Recent Work 1. Visual Sentiment Ontology Step 2 Concept Detector Training and Filtering Good Results Not Good Results

Recent Work 1. Visual Sentiment Ontology Application Live Twitter Stream Sentiment Prediction True stuff. I have mad respect for all the ladies that DO NOT give in to abortion Ouch mr police man #groundzero #hurricanesandy #newjersey

Recent Work 1. Visual Sentiment Ontology Application Live Twitter Stream Sentiment Prediction 2000 tweets with images Two-way (positive/negative) prediction

Recent Work 1. Visual Sentiment Ontology Demo

Thank You! imt.xmu.edu.cn/