1. Subjectivity Annotation Update Josef Ruppenhofer Jan Wiebe

2. Outline Update on our annotations Update on our annotations Exploration of subjectivity and Discourse Treebank annotations Exploration of subjectivity and Discourse Treebank annotations

3. Subjectivity Annotation Before this year: MPQA annotation scheme and corpus Before this year: MPQA annotation scheme and corpus @ www.cs.pitt.edu/mpqa @ www.cs.pitt.edu/mpqa English language versions of articles from the world press (187 news sources) English language versions of articles from the world press (187 news sources) 535 Documents; 11,114 sentences 535 Documents; 11,114 sentences Wiebe, Wilson, Cardie. Annotating Expressions of Opinions and Emotions in Language. LRE 2005.

4. Subjectivity Annotation Current work (goals have been expanded schemes, not high volume annotation) Current work (goals have been expanded schemes, not high volume annotation) Extended MPQA Extended MPQA Additional data annotated Additional data annotated 2005 LRE scheme plus extensions 2005 LRE scheme plus extensions Theresa Wilson’s PhD dissertation (2008) Theresa Wilson’s PhD dissertation (2008) [not-yet-released extensions added to the MPQA corpus] [not-yet-released extensions added to the MPQA corpus] Discourse level relations between opinions Discourse level relations between opinions Subjectivity in health surveillance texts Subjectivity in health surveillance texts Word sense subjectivity and polarity Word sense subjectivity and polarity

5. Extended MPQA Scheme Documents Documents 85 Xbank files 85 Xbank files “Boyan” subset of ULA data “Boyan” subset of ULA data 1/3 completed 1/3 completed expected completion: early summer expected completion: early summer [MPQA] [MPQA]

6. Extended MPQA Scheme Annotators Annotators Josef Josef two undergraduates two undergraduates

7. Training Time & Effort Training Training Josef’s effort: 75 hours (~ 2 weeks) Josef’s effort: 75 hours (~ 2 weeks) 10 preparing materials 10 preparing materials 40 basic training 40 basic training 25 extensions 25 extensions Annotators: 120 hours combined (~1.5 weeks each) Annotators: 120 hours combined (~1.5 weeks each) Problems with scheduling arose (annotators did not work planned hours per week; redundant one-on-one meetings) Problems with scheduling arose (annotators did not work planned hours per week; redundant one-on-one meetings) With perfect scheduling, estimate 1 week to train two annotators (though Josef is involved in production annotation) With perfect scheduling, estimate 1 week to train two annotators (though Josef is involved in production annotation)

8. Production annotation Single annotator per document Single annotator per document Annotator time per document (very rough est) Annotator time per document (very rough est) 2 hours 45 mins 2 hours 45 mins 45 mins of which is time spent on consultation, 15 with each other, 30 with Josef 45 mins of which is time spent on consultation, 15 with each other, 30 with Josef

9. Periodic Agreement Testing documents with known gold standard documents with known gold standard no consultation no consultation ~every 5 documents ~every 5 documents post-mortem meetings (one on one, group) post-mortem meetings (one on one, group) Four annotations to compare (Theresa Wilson, Josef, two undergraduate annotators) Four annotations to compare (Theresa Wilson, Josef, two undergraduate annotators) [Results of previous agreement studies in previous papers] [Results of previous agreement studies in previous papers]

10. Agreement measurement So far, average pair-wise agreement calculated per document So far, average pair-wise agreement calculated per document Full analysis forthcoming Full analysis forthcoming Relative label reliability: agent > direct-subjective > target > attitude > objective-speech-event > expressive-subjective- element Relative label reliability: agent > direct-subjective > target > attitude > objective-speech-event > expressive-subjective- element Given the interactions between the labels, errors are interrelated Given the interactions between the labels, errors are interrelated

11. Annotation Schemes

12. What is Subjectivity? The linguistic expression of somebody’s opinions, sentiments, emotions, evaluations, beliefs, speculations (private states) Private state: state that is not open to objective observation or verification Quirk, Greenbaum, Leech, Svartvik (1985). A Comprehensive Grammar of the English Language.

13. Overview Fine-grained: expression-level rather than sentence or document level Annotate – Subjective xpressions – material attributed to a source, but presented objectively

14. Overview Focus on three ways private states are expressed in language

15. Direct Subjective Expressions Direct mentions of private states The United States fears a spill-over from the anti- terrorist campaign. Private states expressed in speech events “We foresaw electoral fraud but not daylight robbery,” Tsvangirai said.

16. Expressive Subjective Elements [Banfield 1982] “ We foresaw electoral fraud but not daylight robbery, ” Tsvangirai said The part of the US human rights report about China is full of absurdities and fabrications

17. Objective Speech Events Material attributed to a source, but presented as objective fact The government, it added, has amended the Pakistan Citizenship Act 10 of 1951 to enable women of Pakistani descent to claim Pakistani nationality for their children born to foreign husbands.

18. Nested Sources “The US fears a spill-over’’, said Xirao-Nima, a professor of foreign affairs at the Central University for Nationalities. (writer, Xirao-Nima, US) (writer, Xirao-Nima) (writer) “The report is full of absurdities,’’ he continued. (writer, Xirao-Nima) (writer)

19. “The report is full of absurdities,” Xirao-Nina said. Objective speech event anchor: the entire sentence source: implicit: true Direct subjective anchor: said source: intensity: high expression intensity: neutral attitude type: negative target: report Expressive subjective element anchor: full of absurdities source: intensity: high attitude type: negative

20. Objective speech event anchor: the entire sentence source: implicit: true Objective speech event anchor: said source: Direct subjective anchor: fears source: intensity: medium expression intensity: medium attitude type: negative target: [new work] “The US fears a spill-over’’, said Xirao-Nima, a professor of foreign affairs at the Central University for Nationalities.

21. Extensions Wilson 2008 I think people are happy because Chavez has fallen. direct subjective span: are happy source: attitude: inferred attitude span: are happy because Chavez has fallen type: neg sentiment intensity: medium target: target span: Chavez has fallen target span: Chavez attitude span: are happy type: pos sentiment intensity: medium target: direct subjective span: think source: attitude: attitude span: think type: positive arguing intensity: medium target: target span: people are happy because Chavez has fallen

22. Subjectivity Types Wilson 2008 Other (esp. general cognition)

23. Discourse-Level Opinion Frames in Task-Oriented Dialogs (AMI) Frames are defined in terms of their components Frames are defined in terms of their components Opinion spans Opinion spans Opinion type Opinion type Sentiment Sentiment Arguing Arguing Opinion Polarity Opinion Polarity Targets Targets Sources Sources Relationships between targets Relationships between targets Same or alternative Same or alternative Example motivation: polarity and targets interact Example motivation: polarity and targets interact E.g. an argument for one design that is simultaneously an argument against an alternative design E.g. an argument for one design that is simultaneously an argument against an alternative design

24. Subjectivity in Health Surveillance Texts Types Types Sentiment Sentiment Belief Belief Belief about what is the case Belief about what is the case Belief about what should or should not be done Belief about what should or should not be done Knowledge/Awareness of facts Knowledge/Awareness of facts Agreement/Disagreement between sources in the text Agreement/Disagreement between sources in the text … Sources Sources Writer Writer Media Media Non-media organizations Non-media organizations Members of the general public Members of the general public … Targets Targets Occurrence of a disease outbreak Occurrence of a disease outbreak Danger/severity of an outbreak Danger/severity of an outbreak Cause of a disease Cause of a disease Symptoms Symptoms …

25. Exploring the relationship between PDTB (2) and Extended MPQA

26. Penn Discourse TreeBank Hierarchy of discourse relations with 4 top nodes – Temporal – Contingency – Comparison – Expansion

27. Overview Richer interpretations via combination Potential disambiguation both ways

28. A connective is marked as “Restatement” when it indicates that the semantics of Arg2 restates the semantics of Arg1. It is inferred that the situations described in Arg1 and Arg2 hold true at the same time. Expansion Restatement GeneralizationSpecificationEquivalence

29. Subjectivity preserved in a restatement “This means Nestle is now in the candybar business in a big way," said Lisbeth Echeandia, publisher of Orlando, Fla.-based Confectioner Magazine. “For them, it makes all kinds of sense.”

30. Subjectivity preserved in a restatement: PDTB “[This means Nestle is now in the candybar business in a big way ARG1]," said Lisbeth Echeandia, publisher of Orlando, Fla.-based Confectioner Magazine. “IMPLICIT_IN SHORT [For them, it makes all kinds of sense ARG2].

31. Subjectivity preserved in a restatement "This [means Nestle is now in the candybar business in a big way ARGUING-POS]," said Lisbeth Echeandia, publisher of Orlando, Fla.- based Confectioner Magazine. “[For them, it makes all kinds of sense ARGUING-POS]. Related opinions; part of the same larger opinion

32. Subjectivity preserved in a restatement "This [means Nestle is now in the candybar business in a big way ARGUING-POS]," said Lisbeth Echeandia, publisher of Orlando, Fla.- based Confectioner Magazine. “[For them, it makes all kinds of sense ARGUING-POS]. Same polarity, type, source; Hyp: common pattern with restatement

33. Subjectivity preserved in a restatement "This [means Nestle is now in the candybar business in a big way ARGUING-POS]," said Lisbeth Echeandia, publisher of Orlando, Fla.- based Confectioner Magazine. “[For them, it makes all kinds of sense ARGUING-POS]. Semantics of restatement: sameness includes subjectivity

34. Note: Sentiment "This means Nestle is now in the candybar business in a big way," said Lisbeth Echeandia, publisher of Orlando, Fla.-based Confectioner Magazine. “For them, it [makes all kinds of sense SENTIMENT-POS].” Not directly part of the restatement relation

35. The type “Cause” is used when the connective indicates that the situations described in Arg1 and Arg2 are causally influenced and the two are not in a conditional relation … Contingency Cause ReasonResult

36. Subjectivity preserved across Reason relation But Mr. Schwarz welcomes the competition in U.S. Trust's flagship businesses, calling it "flattery." Mr. Schwarz says the competition "broadens the base of opportunity for us." Other firms "are dealing with the masses…”

37. Subjectivity preserved across Reason relation: PDTB [But Mr. Schwarz welcomes the competition in U.S. Trust's flagship businesses ARG1], [calling it "flattery SUP1]." Mr. Schwarz says IMPLICIT_BECAUSE [the competition "broadens the base of opportunity for us ARG2]." Other firms "are dealing with the masses…” ARG2 is a reason for ARG1

38. Subjectivity preserved across Reason relation: subjectivity But Mr. Schwarz [welcomes SENTIMENT-POS] the competition in U.S. Trust's flagship businesses, calling it "flattery." Mr. Schwarz says the competition “[broadens the base of opportunity for us SENTIMENT-POS]." Other firms "are dealing with the masses. Positive evaluation which is a reason for a positive feeling; same overall opinion “I like it because it is so good”

39. Subjectivity preserved across Reason relation: subjectivity But Mr. Schwarz [welcomes SENTIMENT-POS] the competition in U.S. Trust's flagship businesses, calling it "flattery." Mr. Schwarz says the competition “[broadens the base of opportunity for us SENTIMENT-POS]." Other firms "are dealing with the masses. Subjectivity: same source, target, polarity, type; Hyp: common with reason; Help with target recognition, for example.

40. Subjectivity preserved across Reason relation: subjectivity But Mr. Schwarz [welcomes SENTIMENT-POS] the competition in U.S. Trust's flagship businesses, calling it "flattery." Mr. Schwarz says the competition “[broadens the base of opportunity for us SENTIMENT-POS]." Other firms "are dealing with the masses. Semantics of reason: specific subtype, where an evaluation is a reason for an attitude

41. The type “Cause” is used when the connective indicates that the situations described in Arg1 and Arg2 are causally influenced and the two are not in a conditional relation … Contingency Cause ReasonResult

42. Polarity preserved across Result relation Other firms "are dealing with the masses. I don't believe they have the culture" to adequately service high-net-worth individuals, he adds.

43. Polarity preserved across Result relation: PDTB [Other firms "are dealing with the masses ARG1]. I don't believe IMPLICIT_SO [they have the culture" to adequately service high-net- worth individuals ARG2], he adds. ARG2 is a result of ARG1

44. Polarity preserved across Result relation: PDTB [Other firms "are dealing with the masses ARG1]. I don't believe IMPLICIT_SO [they have the culture" to adequately service high-net- worth individuals ARG2], he adds. X said Y: “X said”  X’s belief space “I don’t believe” explicit in second sentence “Swartz said” implicit in first sentence ARG spans: Dis. Rel within Swartz’s belief space

45. Polarity preserved across Result relation: subjectivity Other firms “[are dealing with the masses SENTIMENT-NEG]. I [don't believe they have the culture" to adequately service high-net- worth individuals SENTIMENT-NEG], he adds. Attitude span includes “don’t believe”; schemes require different notions of spans

46. Polarity preserved across Result relation: subjectivity Other firms “[are dealing with the masses SENTIMENT-NEG]. I [don't believe they have the culture" to adequately service high-net- worth individuals SENTIMENT-NEG], he adds. Two negative properties, where the second is a result of the first

47. Polarity preserved across Result relation: subjectivity Other firms “[are dealing with the masses SENTIMENT-NEG]. I [don't believe they have the culture" to adequately service high-net- worth individuals SENTIMENT-NEG], he adds. Dis Rel between ARGS inside his belief space

48. Polarity preserved across Result relation: subjectivity Other firms “[are dealing with the masses SENTIMENT-NEG]. I [don't believe they have the culture" to adequately service high-net- worth individuals SENTIMENT-NEG], he adds. Semantics of result: specific subtype, where a negative state of affairs is the result of another one

49. The class tag “COMPARISON” applies when the connective indicates that a discourse relation is established between Arg1 and Arg2 in order to highlight prominent differences between the two situations. Semantically, the truth of both arguments is independent of the connective or the established relation. Comparison Contrast JuxtapositionOpposition Concession expectation Contra- expectation

50. In that suit, the SEC accused Mr. Antar of engaging in a "massive financial fraud" to overstate the earnings of Crazy Eddie, Edison, N.J., over a three-year period. Through his lawyers, Mr. Antar has denied allegations in the SEC suit and in civil suits previously filed by shareholders against Mr. Antar and others.

51. PDTB [In that suit, the SEC accused Mr. Antar of engaging in a "massive financial fraud" to overstate the earnings of Crazy Eddie, Edison, N.J., over a three- year period. ARG1] IMPLICIT_HOWEVER [ Through his lawyers, Mr. Antar has denied allegations in the SEC suit and in civil suits previously filed by shareholders against Mr. Antar and others. ARG2] Contrast between the SEC accusing Mr. Antar of something, and his denying the accusation

52. Subjectivity In that suit, the SEC [[accused SENTIMENT-NEG] Mr. Antar of engaging in a "massive financial fraud" to overstate the earnings of Crazy Eddie, Edison, N.J. ARGUING-POS], over a three-year period. Through his lawyers, Mr. Antar [has denied AGREE- NEG] allegations in the SEC suit and in civil suits previously filed by shareholders against Mr. Antar and others. Two attitudes combined into one large disagreement between two parties

53. Subjectivity In that suit, the SEC [[accused SENTIMENT-NEG] Mr. Antar of engaging in a "massive financial fraud" to overstate the earnings of Crazy Eddie, Edison, N.J. ARGUING-POS], over a three-year period. Through his lawyers, Mr. Antar [has denied AGREE-NEG] allegations in the SEC suit and in civil suits previously filed by shareholders against Mr. Antar and others. Subjectivity: arguing-pos and agree-neg with different sources; Hyp: common with contrast. Help recognize the implicit contrast.

54. Subjectivity In that suit, the SEC [[accused SENTIMENT-NEG] Mr. Antar of engaging in a "massive financial fraud" to overstate the earnings of Crazy Eddie, Edison, N.J. ARGUING-POS], over a three-year period. Through his lawyers, Mr. Antar [has denied AGREE-NEG] allegations in the SEC suit and in civil suits previously filed by shareholders against Mr. Antar and others. Semantics of comparison: specific case of highlighting prominent differences in attitudes of different people

55. Discourse and opinion relations are not redundant Compare (PDTB) – Its like [people hate him ARG1] because Reason [people love him so much ARG2] and people love him so much because people hate him so much. ARG2 reason for ARG1 – [Some people hate him ARG1]. IMPLICIT Contrast [ Others love him ARG2]. – Subjectivity is the same: contrasting polarities, the same target, different sources.

56. Discourse and opinion relations are not redundant [Some people hate him ARG1]. IMPLICIT Contrast [ Others love him ARG2]. – Sentiment-neg Sentiment-pos – Source and polarity contrasts [I like the Lexus ARG1] but Contrast [my wife likes the Prius ARG2]. – Sentiment-pos Sentiment-pos – Source and target contrasts

57. Discourse and opinion relations are not redundant ARGument and subjectivity spans often do not match exactly

58. Etc. Attributions and nested sources Though the schemes are not redundant, some relations seem to imply subjectivity – E.g., Pragmatic cause; implicit assertions Discourse relations may help uncover inferred attitudes