Cross-Domain Sentiment Tagging using Meta-Classifier and a High Accuracy In-Domain Classifier Balamurali A R*, Debraj Manna~, Pushpak Bhattacharyya~ *IITB-Monash Research Academy, IIT Bombay ~Department of Computer science and Engineering, IIT Bombay ICON-2010, IIT Kharagpur

Roadmap Sentiment Analysis Sentiment classification Problem Solution intuition Approach – H-I System Result Conclusion

What is Sentiment Analysis (SA)? Given a textual portion, – Is the writer expressing sentiment with respect to a topic? – What is that sentiment? Identify the orientation of opinion in a piece of text The movie was fabulous! The movie stars Mr. X The movie was horrible!

Sentiment classification (1/2) – Classification of document(review) based on sentiment content Content being positive/negative – Supervised and unsupervised approaches exist – Supervised approaches perform better I loved CWG opening ceremony I really hate the Indian media for reporting only the bad things happening around in Delhi.  Supervised Approaches: Pang & Lee (2002), Matsumoto et.al(2005) Unsupervised Approaches: Turney (2002), Wan (2008) D1 D2 D1 D2 D3 D4 D5 D4 D5

Sentiment classification (2/2) Sentiment tagging – labeled data generation Issues – Tedious and expensive process – Domain specific nature of Sentiment Analysis – Utility span of labeled data is limited More products with short market life span – Ipod, I-touch, I-Pad Need of people changes – 6 MP camera Good Annotators are difficult to find

Problem we address Problem: A sentiment classifier trained on one domain performs with a lesser accuracy than on a different domain (Aue and Gamon 2005) Solution: Cross-Domain Sentiment Tagging via adaptation – A procedure to use labeled data in an existing domain and use them to create labeled data for new target domains having little or no training data with the help of few hand labeled target data

Intuition Predict the Sentiment Book review: ““As usual, Robin Cook keeps you on the edge of your seat until the end. Excellent reading” From Movie review (Domain Pertinent): ““edge of your seat” Common sense (Domain Independent): “Excellent” Eureka! Its positive!!!!You evolve using the information attained, disregarding the unwanted

Approach (1/3) Step 1: Generate noisy tagged data for the target domain using a Meta-classifier trained on a source domain

Approach (2/3) Step 2 : Select the highly probable correct instances

Approach (3/3) Step 3 : Use them to create a high accuracy in-domain classifier – This classifier then completely classifies the target domain

High Accuracy-Intermediary Tagger System (H-I)

Training Domain Intermediary Tagger System (ITS) Target Domain Tagged Target Domain High Accuracy Classifier (HAC) T1T1 T2T2 Tn Is model optimal? M1M1 M2M2 Mn Labeled Target Data Intermediary Tagged Target Domain No Yes Intermediary Target Domain Models Intermediary Target Domain Data H-I system architecture ITS HAC Selection of Optimal Intermediary Tagged Data Selection of Optimal Intermediary Tagged Data ITS

Source Domain Feature Representation (All Words - Unigrams) Feature Representation (Universal Clue Based) SentiWordNet Base-2 Classifier Model Base-3 Classifier Base-1 Classifier Model Pr (POS)Pr (NEG)Pr (POS)Pr (NEG) SWN (POS) SWN (NEG) Meta Features Meta Classifiers Target domain Training Procedure Intermediary Tagged System (ITS) c Domain Pertinent View Common Sense View Prior Polarity View

ITS - lexicons used SentiWordnet (Esuli and Sebastiani, 2006) – Attaches sentiment score to each synset Positive, negative & objective score Scores sum up to 1 E.g. – {small} – “slight or limited” – pos – 0.125, neg – 0.125, obj – 0.75 Subjectivity Lexicon (Wilson, et al., 2005) – Universal Clues – Consists of manually identified 8000 polar words with their prior polarity – E.g –{wow} – strongly subjective, positive prior polarity

Source Domain Feature Representation (All Words - Unigrams) Feature Representation (Universal Clue Based) SentiWordNet Base-2 Classifier Model Base-3 Classifier Base-1 Classifier Model Pr (POS)Pr (NEG)Pr (POS)Pr (NEG) SWN (POS) SWN (NEG) Meta Features Meta Classifiers Target domain Training Procedure Intermediary Tagged System (ITS) Base-1 Classifier – All words unigram model (All domain pertinent information are recorded using this model) Base-2 Classifier – Classification model based universal clues as features (Generic model) Base- 3 Classifier – Rule based classifier based on SentiWordNet Classification Rule : Document is positive, if prior positive polarity terms >> prior negative polarity terms c

Training Domain Intermediary Tagger System (ITS) Target Domain Tagged Target Domain High Accuracy Classifier (HAC) T1T1 T2T2 Tn Is model optimal? M1M1 M2M2 Mn Labeled Target Data Intermediary Tagged Target Domain No Yes Intermediary Target Domain Models Intermediary Target Domain Data H-I system architecture Selection of Optimal Intermediary Tagged Data Selection of Optimal Intermediary Tagged Data Not all instances of Intermediary Tagged data are correct!

Selection of optimal intermediary tagged data Different sets of intermediary tagged target data created based on different confidence level generated by ITS Different SVM based models created using each set of intermediary tagged target corpus Model tested on few hand labeled target data to find the best confidence threshold This intermediary target set categorized as optimal intermediary tagged data A High Accuracy Classification model is developed using the selected dataset Selected all intermediary tagged data having confidence level 1.> 60%, 2.> 65%

Training Domain Intermediary Tagger System (ITS) Target Domain Tagged Target Domain High Accuracy Classifier (HAC) T1T1 T2T2 Tn Is model optimal? M1M1 M2M2 Mn Labeled Target Data Intermediary Tagged Target Domain No Yes Intermediary Target Domain Models Intermediary Target Domain Data H-I system architecture HAC

High Accuracy Classifier (HAC) Complex features not necessary for creating a High Accuracy Sentiment Classifier Intuition – – In a particular domain, there exist domain specific features (words/phrases) which have high discriminatory power – People use almost similar vocabulary in expressing their sentiment (pertinent to a domain) Process – – An Information gain based feature selection done to select domain specific features – A combination of unigram, bigrams and trigrams used as features Lame movie, Defective kitchen set

Experimental setup Dataset used:- – Multi-Domain Sentiment Dataset (Version 2.0) (Blitzer, et al., 2007) Consists of 4 domain – Book(BS), Dvd(DD), Electronics(ES) and Kitchen(KN). Each domain contains 1000 positive and 1000 negative reviews Models are trained on 1000 positive and 1000 negative data from the source domain. Tested on 950 target data from each class Used 50 positive and 50 negative labeled target data All the experiments are carried out using libSVM (Chih-chung and Chih-Jen, 2001) and Rapidminer 4.6 (Ingo, et al., 2006).

Results - HAC Baseline – All words unigram-based model Combine – uni+bi+trigram model Combine (IG) - uni+bi+trigram model after IG An average increase of 10% in the classification accuracy with respect to the baseline

Results -Top features DOMAINTOP IG BASED FEATURES BOOK (B) waste of, love this, boring, stupid, too many, whatever, ridiculous, two stars DVD (D) worst, horrible, your money, lame, of the best, sucks, barely, ridiculous, save your, is a great, pathetic, dumb, not worth, ruined ELECTRONICS (E) return, terrible, waste your, highly, to return, poor, it back, returning, does not work, do not buy KITCHEN (K) easy to, easy to use. easy to clean, returning, waste your, tried to excellent, defective, horrible, poor, i love it Top IGR based features are different in different domains (Blitzer, et al., 2007). Domain specific nature of Sentiment Analysis

Results - Comparison

Similarity between domains Cosine Similarity Similar domains have high cross classification accuracy BOOKDVDELECTRONICSKITCHEN BOOK DVD ELECTRONICS KITCHEN

Results - Comparison Structural Correspondence Learning (SCL) – State of art for Cross-Domain sentiment classification Dissimilar domains have relatively high increase in accuracy with respect to ITS performance

Results summary

Conclusion Methodology for cross domain sentiment tagging introduced An average cross-domain sentiment classification accuracy of 80% achieved Our system gives a high cross-domain sentiment classification accuracy with an average improvement of 4.39% over the best baseline accuracy

Reference (1/2) Anthony Aue and Michael Gamon,2005,Customizing Sentiment Classifiers to New Domains: A Case Study, Proceedings of Recent Advances in NLP Alina Andreevskaia and Sabine Bergler,2008,When Specialists and Generalists Work Together: Overcoming Domain Dependence in Sentiment Tagging, Proceedings of ACL-08: HLT,PP ,Columbus, Ohio Blitzer, John and Dredze, Mark and Pereira, Fernando, Biographies, Bollywood, Boom-boxes and Blenders: Domain Adaptation for Sentiment Classification,Proceedings of the 45th Annual Meeting of ACL,2007,pages = ,Prague, Czech Republic Andrea Esuli and Fabrizio Sebastiani,SentiWordNet: A Publicly Available Lexical Resource for Opinion Mining,Proceedings of LREC- 06,2006,Genova, Italy Theresa Wilson and Janyce Wiebe and Paul Hoffmann,Recognizing Contextual Polarity in Phrase-Level Sentiment Analysis,Proceedings of EMNLP,2005,pages = ,Vancouver, Canada Bo Pang and Lillian Lee and Shivakumar Vaithyanathan,Thumbs up? Sentiment Classification Using Machine Learning Techniques,Proceedings of EMNLP,2002,pages = 79-86,Philadelphia, Pennsylvania David H. Wolpert,Stacked generalization,Neural Networks,Volume 5, 1992, Pages= Chih-Chung Chang and Chih-Jen Lin,LIBSVM: a library for support vector machines,2001 Wu, Ting-Fan and Lin, Chih-Jen and Weng, Ruby C.,Probability Estimates for Multi-class Classification by Pairwise Coupling,Journal of Machine Learning and research,Vol 5,2004},

Reference (2/2) Bo Pang and Lillian Lee and Shivakumar Vaithyanathan,Thumbs up? Sentiment Classification Using Machine Learning Techniques,Proceedings of EMNLP,2002,pages = 79-86,Philadelphia, Pennsylvania David H. Wolpert,Stacked generalization,Neural Networks,Volume 5, 1992, Pages= Chih-Chung Chang and Chih-Jen Lin,LIBSVM: a library for support vector machines,2001 Wu, Ting-Fan and Lin, Chih-Jen and Weng, Ruby C.,Probability Estimates for Multi-class Classification by Pairwise Coupling,Journal of Machine Learning and research,Vol 5,2004},975—1005 Read, Jonathon,Using emoticons to reduce dependency in machine learning techniques for sentiment classification, ACL '05: Proceedings of the ACL Student Research Workshop},2005,pages =43--48,Annrbor, Michigan,Morristown, NJ, USA Bo Pang and Lillian Lee, 2005, Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales, Proceedings of the ACL. Wiebe, J., Wilson, T., Bruce, R., Bell, M. & Martin, M.,Learning Subjective Language, 2004,Computational Linguistics. Philip J. Stone, Dexter C. Dunphy, Marshall S. Smith, and Daniel M. Ogilvie. The General Inquirer: A Computer Approach to Content Analysis. MIT Press, Taboada, M. and J. Grieve (2004) Analyzing Appraisal Automatically. American Association for Artificial Intelligence Spring Symposium on Exploring Attitude and Affect in Text. Stanford. March AAAI Technical Report SS (pp ).

Meta-Classifier (1/2) An ensemble classifier – Use of multiple models to obtain better predictive performance than using just a constituent model Commonly known as stacking – Base models are called level-0 models – Level -1 model combines level-0 predictions If only two level – final level features are called meta- features A classifier modeled on meta-features to get final prediction

Meta-Classifier (2/2) Motivation – Reduce bias (model) A combination of multiple classifiers may learn a more expressive concept class than a single classifier Key step Formation of an meta-classifier of diverse classifiers from a single training set

Extra slide -Information gain A term goodness criteria – how good is the term for classification – IG(C|X) = How many bits on average would it save me if both ends of the line knew X? – IG(X=t) = H(C) - H(C|X=t) How to calculate t = term to be considered P(ci) – probability of class P(t) – probability of term occurring P(t’)- probability of term not occurring