1. SentiStrength: Sentiment Strength Detection in MySpace and Twitter Mike Thelwall Statistical Cybermetrics Research Group University of Wolverhampton, UK Virtual Knowledge Studio (VKS) Information Studies

2. SentiStrength Objective 1. Detect positive and negative sentiment strength in short informal text 1. Develop workarounds for lack of standard grammar and spelling 2. Harness emotion expression forms unique to MySpace or CMC (e.g., :-) or haaappppyyy!!!) 3. Classify simultaneously as positive 1-5 AND negative 1-5 sentiment 2. Apply to MySpace comments and social issues

3. SentiStrength Algorithm - Core List of 890 positive and negative sentiment terms and strengths (1 to 5), e.g. ache = -2, dislike = -3, hate=-4, excruciating -5 encourage = 2, coolest = 3, lover = 4 Sentiment strength is highest in sentence; or highest sentence if multiple sentences

4. Examples My legs ache. You are the coolest. I hate Paul but encourage him. -2 3 -4 2 1, -2 positive, negative 3, -1 2, -4

5. Term Strength Optimisation Term strengths (e.g., ache = -2) initially fixed by human coder Term strengths optimised on training set with 10-fold cross-validation Adjust term strengths to give best training set results then evaluate on test set E.g., training set: “My legs ache”: coder sentiment = 1,-3 => adjust sentiment of “ache” from -2 to -3.

6. SentiStrength Algorithm -Extra Spelling correction for repeated letters Helllllo -> Hello (emphasis: llll) Tagging approach used (see next slide) Extra heuristics Emphasis acts to enhance + or – emotion Emotion words ignored in questions Take strongest positive or negative expression in whole comment Booster words (e.g., very, some)

7. Tagging HIIIIII MY MATE!!!!!!!! HIIIIII MY MATE !!!!!!!! HI MY MATE! 2 3 Overall 3, -1 mate = 2

8. Experiments Development data = 2600 MySpace comments coded by 1 coder Test data = 1041 MySpace comments coded by 3 independent coders Comparison against a range of standard machine learning algorithms

9. Inter-coder agreement Comparison+ve agree- ment -ve agree- ment Coder 1 vs. 251.0%67.3% Coder 1 vs. 355.7%76.3% Coder 2 vs. 361.4%68.2% Krippendorff’s inter-coder weighted alpha = 0.5743 for positive and 0.5634 for negative sentiment Only moderate agreement between coders but it is a hard 5-category task

10. Machine learning methods +ve

11. Machine learning methods -ve

12. Results:+ve sentiment strength AlgorithmOpt. Feat. Accu- racy Acc. +/- 1 class Corr.Mean % abs. error SentiStrength-60.6%96.9%.59922.0% Simple logistic regression70058.5%96.1%.55723.2% SVM (SMO)80057.6%95.4%.53824.4% J48 classification tree70055.2%95.9%.54824.7% JRip rule-based classifier70054.3%96.4%.47628.2% SVM regression (SMO)10054.1%97.3%.46928.2% AdaBoost10053.3%97.5%.46428.5% Decision table20053.3%96.7%.43128.2% Multilayer Perceptron10050.0%94.1%.42230.2% Naïve Bayes10049.1%91.4%.56727.5% Baseline-47.3%94.0%-31.2% Random-19.8%56.9%.01682.5%

13. Results:-ve sentiment strength AlgorithmOpt. feat. AccuracyAcc. +/- 1 class Corr.Mean % absolute error SVM (SMO)10073.5%92.7%.42116.5% SVM regression (SMO)30073.2%91.9%.36317.6% Simple logistic regression80072.9%92.2%.36417.3% SentiStrength-72.8%95.1%.56418.3% Decision table10072.7%92.1%.34617.0% JRip rule-based classifier50072.2%91.5%.30917.3% J48 classification tree40071.1%91.6%.23518.8% Multilayer Perceptron10070.1%92.5%.34620.0% AdaBoost10069.9%90.6%-16.8% Baseline-69.9%90.6%-16.8% Naïve Bayes20068.0%89.8%.31127.3% Random-20.5%46.0%.010157.7%

14. SentiStrength Components Type% Consecutive +ve words not used as boosters61.2 Emoticons ignored61.2 Negating words not switch (e.g., not happy)61.0 SentiStrength standard configuration60.9 Booster words ignored (e.g., very)60.7 Automatic spelling correction disabled60.6 Exclamation marks not given a strength of 260.6 Extra multiple letters not used as boosters60.4 Neutral words with emphasis not counted as +ve60.1 SentiStrength with all the above changes57.5

15. Example differences/errors THINK 4 THE ADD Computer (1,-1), Human (2,-1) 0MG 0MG 0MG 0MG 0MG 0MG 0MG 0MG!!!!!!!!!!!!!!!!!!!!N33N3R!!!!!!!!!!!!!!!! Computer (2,-1), Human (5,-1)

16. Selected variations tested Modification (for positive sentiment) Accuracy +/- 1 class corr.Mean Abs. % err. Negating words not used to switch following sentiment (e.g., not happy) 60.87%97.50%.620621.28% SentiStrength standard algorithm 60.64%96.90%.598621.96% Exclamation marks not given a strength of 2 60.51%96.62%.603521.47% Automatic spelling correction disabled 60.39%96.88%.596122.05% Extra multiple letters not used as emotion boosters 60.21%96.81%.595222.16% Neutral words with emphasis not counted as positive emotion 60.13%96.79%.596621.90% SentiStrength with no extras 57.44%96.07%.607321.91%

17. Application - Evidence of emotion homophily in MySpace Automatic analysis of sentiment in 2 million comments exchanged between MySpace friends Correlation of 0.227 for +ve emotion strength and 0.254 for –ve People tend to use similar but not identical levels of emotion to their friends in messages

18. CYBEREMOTIONS = data gathering + complex systems methods + ICT outputs Collective Emotions in Cyberspace Sentistrength

19. Application – sentiment in Twitter events Analysis of a corpus of 1 month of English Twitter posts Automatic detection of spikes (events) Sentiment strength classification of all posts Assessment of whether sentiment strength increases during important events Result – negative sentiment normally increases, positive sentiment might tend to increase

20. Automatically-identified Twitter spikes

21. Chile

22. Hawaii

23. #oscars

24. Tiger Woods

26. Conclusion Automatic classification of emotion on a 5 point positive and negative scale seems possible for MySpace …And other similar short computer text messages? Hard to get accuracy much over 60%? Next = analyse emotion in online debates

27. Publication Thelwall, M., Buckley, K., Paltoglou, G. Cai, D., & Kappas, A. (in press). Sentiment strength detection in short informal text. Journal of the American Society for Information Science and Technology.Sentiment strength detection in short informal text Thelwall, M., Wilkinson, D. & Uppal, S.(2010). Data mining emotion in social network communication: Gender differences in MySpace, Journal of the American Society for Information Science and Technology, 61(1), 190-199.Data mining emotion in social network communication: Gender differences in MySpace