1. Exploiting Subjectivity Classification to Improve Information Extraction Ellen Riloff University of Utah Janyce Wiebe University of Pittsburgh William Phillips University of Utah

2. Subjectivity ? Definition: Subjective language expresses or refers to opinions, emotions, sentiments and other private states. Related Work: –Sentiments (Turney & Littman 2003; Dave, Lawrence, & Pennock 2003; Pang & Lee 2004) –Product Reputation Tracking (Morinaga et al. 2002; Yi et al. 2003) –Opinion Oriented Summarization and QA (Hu & Liu 2004; Yu & Hatzivassiloglou 2003) Opinion - personal beliefs Emotion - state of mind Sentiments - positive/negative judgements

3. Motivation Our observation: many false hits produced by Information Extraction (IE) systems come from subjective sentences. Hypothesis: we can improve IE performance by avoiding extractions from subjective sentences.

4. Examples “D’Aubruisson unleashed harsh attacks on Duarte…” “The Parliament exploded into fury against the government when word leaked out…” “The subversives must suspend the aggression against the people and the destruction of the economy…”

5. The Big Picture Subjective Sentence Classifier subjective sentences objective sentences Full Information Extraction Selective Information Extraction

6. The Subjectivity Classifier Most documents contain a mix of subjective and objective sentences –44% of sentences in newspaper articles subjective! (Wiebe et al. 2004) We used the Naïve Bayes subjective sentence classifier developed by Wiebe & Riloff [2005]. –Classifies at sentence level –unsupervised –rivals best supervised methods

7. Initial Training Data Creation rule-based subjective sentence classifier rule-based objective sentence classifier subjective & objective sentences unlabeled texts subjective clues

8. Naïve Bayes training POS features subjective clues Naïve Bayes Training extraction pattern learner training set objective patterns subjective patterns Naïve Bayes Classifier

9. NB Confidence Measure CM =

10. MUC-4 IE Task To extract information about terrorist events in Latin America. Evaluated performance on 4 types of information: –perpetrators (individuals), victims, targets, weapons Corpus: 1700 texts –1400 used for training, 100 for tuning, 200 for testing Used Autoslog-TS to generate extraction patterns –system used 397 patterns

11. Base IE System Performance SystemRec Prec F #Correct #Wrong IE.52.42.47266 367

12. Filtering Subjective Sentences SystemRec Prec F #Correct #Wrong IE.52.42.47266 367 IE+SubjFilter.44.44.44218 (-48) 273 (-94)

13. Source Attribution Sentences In news articles, factual information is often prefaced with a source attribution. Examples: “The Associated Press reported…”“The President stated…” Source attribution sentences often contain important facts even if subjective language is also present.

14. Source Attribution Modification Keep the subjective sentences if they contain a source attribution. 1) the sentence contains a communication verb: {affirm, announce, cite, confirm, convey, disclose, report, tell, say, state } 2) the subjectivity classifier considers the sentence to be only weakly subjective (CM  25)

15. Results with Source Attribution Modification SystemRec. Prec. F #Correct #Wrong IE.52.42.47266 367 IE+SubjFilter.44.44.44218(-48) 273(-94) IE+SubjFilter2.46.44.45231(-35) 289(-78)

16. Selective Filtering We observed that subjective sentence can contain important facts. For example: “He was outraged by the terrorist attack on the World Trade Center.” Modification: selectively extract information from subjective sentences Done using Indicator Patterns.

17. Indicator Patterns We defined an indicator pattern as a pattern that has the following Autoslog-TS statistics : P(relevant | pattern)  0.65 and Frequency  10 Indicator Patterns clearly represent a fact of interest –“murder of X” – “X was assassinated”.

18. Results for Selective Subjectivity Filtering SystemRec Prec F #Correct #Wrong IE.52.42.47266 367 IE+SubjFilter.44.44.44218 (-48) 273 (-94) IE+SubjFilter2.46.44.45231 (-35) 289 (-78) IE+SF2+Slct.51.45.48258 (-8) 311 (-56)

19. Removing Subjective Extraction Patterns Example: “….to destroy the building.” “…to destroy the process of reconciliation.” Use subjectivity analysis to remove subjective patterns. We classified a pattern as subjective if: 1) P(subjective | pattern) >.50 and 2) frequency  10

20. Final Results SystemRec Prec F #Correct #Wrong IE.52.42.47266 367 IE+SubjFilter.44.44.44218 (-48) 273 (-94) IE+SubjFilter2.46.44.45231 (-35) 289 (-78) IE+SF2+Slct.51.45.48258 (-8) 311 (-56) IE+SF2+Slct -SubjEPs.51.46.48258(-8) 305(-62)

21. Subjectivity Filtering Combined with Topic Classification SystemRec Prec IE.52.42 IE w/Perfect TC.52.53 IE w/Perfect TC + SubjFilter.51.56

22. Conclusions Subjectivity filtering strategies improved IE precision with minimal recall loss. The benefits of subjectivity classification are synergistic with those of topic classification. As subjectivity classification improves, we expect corresponding improvements to IE.

23. IE Evaluation  Performed at extraction level, before template generation Standard IE System textsextracts Slot Extraction Component Template Generation Component

24. We defined an indicator pattern as a pattern that has the following Autoslog-TS statistics : P(relevant | pattern)  0.65 and Frequency  10 Using only the indicator patterns for IE not sufficient. Rec Prec F IE.52.42.47 IE (Indicators Only).40.54.46

25. IE System We used Autoslog-TS to generate extraction patterns. –40,553 distinct patterns were learned We manually reviewed top patterns (2808 patterns) The final system used 397 patterns.

26. Examples of Filtered Extractions The demonstrators, convoked by the solidarity with Latin America Committee, verbally attacked Salvadoran President Alfredo Cristiani and have asked the Spanish government to offer itself as a mediator to promote and end to the armed conflict. PATTERN: attacked VICTIM: “Salvadoran President Alfredo Cristiani”

27. Examples of Filtered Extractions The crime was directed at hindering the development of the electoral process and destroying the reconciliation process… PATTERN: destroying TARGET: “the reconciliation process” Presidents, political and social figures of the continent have said that the solution is not based on the destruction of a native plant but in active fight against drug consumption. PATTERN: destruction of TARGET: “a native plant”

28. Breakdown by Extraction Type Category BaselineSubjFilter Rec PrecRec Prec Perp.47.33.45.38 Victim.51.50.50.52 Target.63.42.62.47 Weapon.45.39.43.42 Total.52.42.51.46

29. Subjective Patterns attacks on to attack communique by to destroy was linkedleaders of unleashedwas aimed at offensive against dialogue with The following extraction patterns were classified as subjective:

30. Metaphor False hits can come from subjective sentences that contain metaphorical language. The Parliament exploded into fury against the government when word leaked out…