Automatic Identification of Pro and Con Reasons in Online Reviews Soo-Min Kim and Eduard Hovy USC Information Sciences Institute Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions 2015/11/241

Outline 1.Introduction 2.Pros and Cons in Online Reviews 3.Finding Pros and Cons 4.Data 5.Experiments and Results 6.Conclusions and Future work 2015/11/242

1. Introduction (1/3) This paper is focus on a critical problem of opinion analysis, identifying reasons for opinions, especially for opinions in online product reviews “What are the reasons that the author of this review likes or dislikes the product?” In hotel reviews, more useful information would be “This hotel is great for families with young infants” or “Elevators are grouped according to floors, which makes the wait short” 2015/11/243

1. Introduction (2/3) This paper focus on extracting pros and cons which include not only sentences that contain opinion-bearing expressions about products and features but also sentences with reasons It creates duplicate files. Video drains battery. It won't play music from all music stores 2015/11/244

1. Introduction (3/3) Labeling each sentence is a time-consuming and costly task They propose a framework for automatically identifying reasons in online reviews and introduce a novel technique to automatically label training data for this task Assume reasons in an online review document are closely related to pros and cons represented in the text They use those pros and cons of reviews extract from epinion.com to automatically label sentences in the reviews on which subsequently train the classification system 2015/11/245

2. Pros and Cons in Online Reviews In general, researchers study opinion at three different levels: word level, sentence level, and document level Many researchers consider a whole document as the unit of an opinion to be too coarse This study take the approach that a review text has a main about a given product, but also includes various reasons for recommendation or non-recommendation 2015/11/246

3. Finding Pros and Cons 2015/11/247

3.1 Automatically Labeling Pro and Con Sentences (1/2) epinions.com explicitly state pros and cons phrases in their respective categories by each review’s author along with the review text Collecting a large set of triplets from epinions.com The system extracts comma-delimited phrases from each pro and con field, generating two sets of phrases: {P1, P2, …, Pn} for pros and {C1, C2, …, Cm} for cons Compare these phrases to the sentences in the text in the “Full Review”. For each phrase in {P1, P2, …, Pn} and {C1, C2, …, Cm}, the system checks each sentence to find a sentence that covers most of the words in the phrase Annotate this sentence with the appropriate “pro” or “con” label All remaining sentences with neither label are marked as “neither” 2015/11/248

3.1 Automatically Labeling Pro and Con Sentences (2/2) 2015/11/249

3.2 Modeling with Maximum Entropy Classification (1/2) Maximum Entropy classification is used for the task of finding pro and con sentences in a given review They separated the task of finding pro and con sentences into two phases, each being a binary classification – Identification phase: separates pro and con candidate sentences (CR and PR) from sentences irrelevant to either of them (NR) – Classification phase: classifies candidates into pros (PR) and cons (CR) 2015/11/2410

3.2 Modeling with Maximum Entropy Classification (2/2) Model the conditional probability of a class c given a feature vector x as follows: is a normalization factor is a feature function which has a binary value, 0 or 1 is a weight parameter for the feature function and higher value of the weight indicates that is an important feature for a class c 2015/11/2411

3.3 Features (1/2) Three types of features: lexical features, positional features, and opinion- bearing word features For lexical features, they investigate the intuition that there are certain words that are frequently used in pro and con sentences which are likely to represent reasons why an author writes a review, ex. “because” and “that’s why” For positional features, these features test the intuition used in document summarization that important sentences that contain topics in a text have certain positional patterns in a paragraph For opinion-bearing word features – derived a list of opinion-bearing words from a large news corpus by separating opinion articles such as letters or editorials from news articles which simply reported news or events – calculated semantic orientations of words based on WordNet synonyms 2015/11/2412

3.3 Features (2/2) 2015/11/2413

4. Data (1/3) Two different sources: epinions.com and complaints.com Data from epinions.com is mostly used to train the system Data from complaints.com is to test how the trained model performs on new data 2015/11/2414

4. Data (2/3) Dataset 1: Automatically Labeled Data Two different domains of reviews from epinions.com: product reviews and restaurant reviews The purpose of selecting one of electronics products and restaurants as topics of reviews for our study is to test our approach in two extremely different situations Number of reviews Number of sentences Average numbers of sentences Product reviews (mp3 players) Restaurant reviews /11/2415

4. Data (3/3) Dataset 2: Complaints.com Data From the database in complaints.com 59 complaints reviews about mp3 players and 322 reviews about restaurants They tested our system on this dataset and compare the results against human judges’ annotation results 2015/11/2416

5. Experiments and Results Two goals in the experiments: – Investigate how well the pro and con detection model with different feature combinations performs on the data we collected from epinions.com – See how well the trained model performs on new data from a different source, complaint.com Data are divided into 80% for training, 10% for development, and 10% for test They measure the performance with accuracy (Acc), precision (Prec), recall (Recl), and F-score 2015/11/2417

5.1 Experiments on Dataset 1 (1/4) Identification step: The baseline system assigned all sentences as reason and achieved 57.75% and 54.82% of accuracy 2015/11/2418

5.1 Experiments on Dataset 1 (2/4) The system achieved a very low score when it only used opinion word features – Pro and con sentences in reviews are often purely factual – opinion features improved both precision and recall when combined with lexical features in restaurant reviews Experiments on mp3 players reviews achieved mostly higher scores than restaurants – frequently mentioned keywords of product features (e.g. durability) may have helped performance, especially with lexical features The positional features did not help much for this task 2015/11/2419

5.1 Experiments on Dataset 1 (3/4) Classification step: 2015/11/2420

5.1 Experiments on Dataset 1 (4/4) The baseline system marked all sentences as pros and achieved 53.87% and 50.71% accuracy for each domain Unlike the identification task, opinion words by themselves achieved the best accuracy in both mp3 player and restaurant domains opinion words played more important roles in classifying pros and cons than identifying them Position features helped recognizing con sentences in mp3 player reviews 2015/11/2421

5.2 Experiments on Dataset 2 (1/3) Dataset 2 from complaints.com has no training data, we trained a system on Dataset 1 and applied it to Dataset 2 Gold Standard Annotation: – Four humans annotated 3 sets of test sets: Testset 1 with 5 complaints (73 sentences), Testset 2 with 7 complaints (105 sentences), and Testset 3 with 6 complaints (85 sentences) – Testset 1 and 2 are from mp3 player complaints and Testset 3 is from restaurant reviews – Each test set was annotated by 2 humans – The average pair-wise human agreement was 82.1% 2015/11/2422

5.2 Experiments on Dataset 2 (2/3) The goal is to identify reason sentences in complaints Assume each annotator’s answers separately as a gold standard 2015/11/2423

5.2 Experiments on Dataset 2 (3/3) Some examples of sentences that our system identified as reasons of complaints (1) Unfortunately, I find that I am no longer comfortable in your establishment because of the unprofessional, rude, obnoxious, and unsanitary treatment from the employees. (2) They never get my order right the first time and what really disgusts me is how they handle the food. (3) The kids play area at Braum's in The Colony, Texas is very dirty. (4) The only complaint that I have is that the French fries are usually cold. (5) The cashier there had short changed me on the payment of my bill. 2015/11/2424

6. Conclusions and Future Work This paper proposes a framework for identifying a critical element of reviews to answer the question, “What are reasons that the author of a review likes or dislikes the product?” They present a novel technique that automatically labels a large set of pro and con sentences using clue phrases for pros and cons in epinions.com in order to train our system In the future, extend the identification system on other sorts of opinion texts, such as debates about political and social agenda that can found on blogs or news group discussions 2015/11/2425