SentiStrength: Sentiment Strength Detection in MySpace and Twitter Mike Thelwall Statistical Cybermetrics Research Group University of Wolverhampton, UK Virtual Knowledge Studio (VKS) Information Studies

SentiStrength Objective 1. Detect positive and negative sentiment strength in short informal text 1. Develop workarounds for lack of standard grammar and spelling 2. Harness emotion expression forms unique to MySpace or CMC (e.g., :-) or haaappppyyy!!!) 3. Classify simultaneously as positive 1-5 AND negative 1-5 sentiment 2. Apply to MySpace comments and social issues

SentiStrength Algorithm - Core List of 890 positive and negative sentiment terms and strengths (1 to 5), e.g. ache = -2, dislike = -3, hate=-4, excruciating -5 encourage = 2, coolest = 3, lover = 4 Sentiment strength is highest in sentence; or highest sentence if multiple sentences

Examples My legs ache. You are the coolest. I hate Paul but encourage him , -2 positive, negative 3, -1 2, -4

Term Strength Optimisation Term strengths (e.g., ache = -2) initially fixed by human coder Term strengths optimised on training set with 10-fold cross-validation Adjust term strengths to give best training set results then evaluate on test set E.g., training set: “My legs ache”: coder sentiment = 1,-3 => adjust sentiment of “ache” from -2 to -3.

SentiStrength Algorithm -Extra Spelling correction for repeated letters Helllllo -> Hello (emphasis: llll) Tagging approach used (see next slide) Extra heuristics Emphasis acts to enhance + or – emotion Emotion words ignored in questions Take strongest positive or negative expression in whole comment Booster words (e.g., very, some)

Tagging HIIIIII MY MATE!!!!!!!! HIIIIII MY MATE !!!!!!!! HI MY MATE! 2 3 Overall 3, -1 mate = 2

Experiments Development data = 2600 MySpace comments coded by 1 coder Test data = 1041 MySpace comments coded by 3 independent coders Comparison against a range of standard machine learning algorithms

Inter-coder agreement Comparison+ve agree- ment -ve agree- ment Coder 1 vs %67.3% Coder 1 vs %76.3% Coder 2 vs %68.2% Krippendorff’s inter-coder weighted alpha = for positive and for negative sentiment Only moderate agreement between coders but it is a hard 5-category task

Machine learning methods +ve

Machine learning methods -ve

Results:+ve sentiment strength AlgorithmOpt. Feat. Accu- racy Acc. +/- 1 class Corr.Mean % abs. error SentiStrength-60.6%96.9% % Simple logistic regression %96.1% % SVM (SMO) %95.4% % J48 classification tree %95.9% % JRip rule-based classifier %96.4% % SVM regression (SMO) %97.3% % AdaBoost %97.5% % Decision table %96.7% % Multilayer Perceptron %94.1% % Naïve Bayes %91.4% % Baseline-47.3%94.0%-31.2% Random-19.8%56.9% %

Results:-ve sentiment strength AlgorithmOpt. feat. AccuracyAcc. +/- 1 class Corr.Mean % absolute error SVM (SMO) %92.7% % SVM regression (SMO) %91.9% % Simple logistic regression %92.2% % SentiStrength-72.8%95.1% % Decision table %92.1% % JRip rule-based classifier %91.5% % J48 classification tree %91.6% % Multilayer Perceptron %92.5% % AdaBoost %90.6%-16.8% Baseline-69.9%90.6%-16.8% Naïve Bayes %89.8% % Random-20.5%46.0% %

SentiStrength Components Type% Consecutive +ve words not used as boosters61.2 Emoticons ignored61.2 Negating words not switch (e.g., not happy)61.0 SentiStrength standard configuration60.9 Booster words ignored (e.g., very)60.7 Automatic spelling correction disabled60.6 Exclamation marks not given a strength of Extra multiple letters not used as boosters60.4 Neutral words with emphasis not counted as +ve60.1 SentiStrength with all the above changes57.5

Example differences/errors THINK 4 THE ADD Computer (1,-1), Human (2,-1) 0MG 0MG 0MG 0MG 0MG 0MG 0MG 0MG!!!!!!!!!!!!!!!!!!!!N33N3R!!!!!!!!!!!!!!!! Computer (2,-1), Human (5,-1)

Selected variations tested Modification (for positive sentiment) Accuracy +/- 1 class corr.Mean Abs. % err. Negating words not used to switch following sentiment (e.g., not happy) 60.87%97.50% % SentiStrength standard algorithm 60.64%96.90% % Exclamation marks not given a strength of %96.62% % Automatic spelling correction disabled 60.39%96.88% % Extra multiple letters not used as emotion boosters 60.21%96.81% % Neutral words with emphasis not counted as positive emotion 60.13%96.79% % SentiStrength with no extras 57.44%96.07% %

Application - Evidence of emotion homophily in MySpace Automatic analysis of sentiment in 2 million comments exchanged between MySpace friends Correlation of for +ve emotion strength and for –ve People tend to use similar but not identical levels of emotion to their friends in messages

CYBEREMOTIONS = data gathering + complex systems methods + ICT outputs Collective Emotions in Cyberspace Sentistrength

Application – sentiment in Twitter events Analysis of a corpus of 1 month of English Twitter posts Automatic detection of spikes (events) Sentiment strength classification of all posts Assessment of whether sentiment strength increases during important events Result – negative sentiment normally increases, positive sentiment might tend to increase

Automatically-identified Twitter spikes

Chile

Hawaii

#oscars

Tiger Woods

Conclusion Automatic classification of emotion on a 5 point positive and negative scale seems possible for MySpace …And other similar short computer text messages? Hard to get accuracy much over 60%? Next = analyse emotion in online debates

Publication Thelwall, M., Buckley, K., Paltoglou, G. Cai, D., & Kappas, A. (in press). Sentiment strength detection in short informal text. Journal of the American Society for Information Science and Technology.Sentiment strength detection in short informal text Thelwall, M., Wilkinson, D. & Uppal, S.(2010). Data mining emotion in social network communication: Gender differences in MySpace, Journal of the American Society for Information Science and Technology, 61(1), Data mining emotion in social network communication: Gender differences in MySpace