Pollyanna Gonçalves (UFMG, Brazil) Matheus Araújo (UFMG, Brazil) Fabrício Benevenuto (UFMG, Brazil) Meeyoung Cha (KAIST, Korea) Comparing and Combining Sentiment Analysis Methods

 Key component of a new wave of applications that explore social network data  Summary of public opinion about:  politics, products, services (e.g. a new car, a movie), etc.  Monitor social network data (in real-time)  Common as polarity analysis (positive or negative) Sentiment Analysis on Social Networks

 Which method to use?  There are several methods proposed for different contexts  There are several popular methods  Validations based on examples, comparisons with baseline, with use of limited datasets  There is not a proper comparison among methods  Advantages? Disadvantages? Limitations? Sentiment Analysis Methods

 Compare 8 popular sentiment analysis methods  Focus on the task of detecting polarity: positive vs. negative  Combine methods  Deploy the methods in a system This talk

Ifeel System & Conclusions Methods & Methodology Comparing & Combining

 Extracted from instant messages services  Skype, MSN, Yahoo Messages, etc.  Grouped as positive and negative Emoticons

 Lexical method (paid software)  Allows to optimize the lexical dictionary -> we used the default  Measures various emotional, cognitive, and structural components  We only consider sentiment-relevant categories such as positivity, negativity Linguistic Inquiry and Word Count (LIWC)

 Lexical approach based on the WordNet dictionary  Groups words in synonyms  Detects positivity, negativity, and neutrality of texts SentiWordNet

 Lexical method adapted from a psychometric scale  Consists of a dictionary of adjectives associated to sentiments  Positive: Joviality, assurance, serenity, and surprise  Negative: Fear, sadness, guilt, hostility, shyness and fatigue PANAS-t

 Uses a well-known lexical dictionary namely Affective Norms for English Words (ANEW)  Produces a scale of happiness  1 (extremely happy) to 9 (extremely unhappy)  We consider [1..5) for negative and [5..9] for positive Happiness Index

 Combines 9 supervised machine learning methods  Estimates the strength of positive and negative sentiment in a text  We used the trained model provided by the authors SentiStrengh

 Machine learning method, trained with Naïve Bayes’ model  Trained model implemented as a python library  Classify tweets in JSON format for positive, negative, neutral and unsure SAIL/AIL Sentiment Analyzer (SASA)

 Extract cognitive and affective information using natural language processing techniques  Uses the affective categorization model Hourglass of Emotions  Provides an approach that classify messages as positive and negative SenticNet

 Comparison of coverage and prediction performance across different datasets  Dataset 1: human labeled  About 12,000 messages labeled with Amazon Mechanical Turk:  Twitter, MySpace, YouTube and Digg comments, BBC and Runners World forums  Dataset 2: unlabeled  Complete snapshot from Twitter (collected in 2009) ~2 billion tweets  Extracted tragedies, disasters, movie releases, and political events  Focus on the English messages Methodology

Ifeel System & Conclusions Methods & Methodology Comparing & Combining

What is the coverage of each method?

Coverage vs. Prediction Performance  Emoticons: best prediction and worst coverage  SentiStrenght: second in prediction and third in coverage

Prediction Performance across datasets TwitterMySpaceYoutubeBBCDiggRunners World PANAS-t Emoticons SASA SenticNet SentiWordNet SentiStrength Happiness Index LIWC  Strong variations across datasets

Prediction Performance across datasets TwitterMySpaceYoutubeBBCDiggRunners World PANAS-t Emoticons SASA SenticNet SentiWordNet SentiStrength Happiness Index LIWC  Worst performance for datasets containing formal text

Polarity Analysis Detected only positive Sentiments!  Methods tend to detect more positive sentiments  Positive as positive is usually greater than negative as negative Even disasters were classified predominantly as positive

 Combines 7, of the 8 methods analyzed  Emoticons, SentiStrength, Happiness Index, SenticNet, SentiWordNet, PANAS-t, SASA  Removed LIWC (paid method)  Weights are distributed according to the rank of prediction performance:  Higher weight for the method with highest F-measure  Emoticon received weight 7 and PANAS-t 1 Combined Method

 Best coverage and second in prediction performance  4 methods combined are sufficient

Ifeel System & Conclusions Methods & Methodology Comparing & Combining

 Example for:  “Feeling too happy today :)“  Deploys all methods, except LIWC  Allows to evaluate an entire file  Allows to change parameters on the methods iFeel (Beta version)

 We compare 8 popular sentiment analysis methods for detecting polarity  No method had the best results in all analysis  Prediction performance largely varies according to the dataset  Most methods are biased towards positivity  We propose a combined method  Achieves high coverage and high prediction performance  Ifeel: methods deployed and easily available  Future work: Compare others methods like POMS and EMOLEX Conclusions

Questions? Thank you!