1. 1 NSF-ULA Sense tagging and Eventive Nouns Martha Palmer, Miriam Eckert, Jena D. Hwang, Susan Windisch Brown, Dmitriy Dligach, Jinho Choi, Nianwen Xue University of Colorado Departments of Linguistics and Computer Science Institute for Cognitive Science

2. 2 OntoNotes Participating Sites: BBN University of Colorado University of Pennsylvania USC/ISI http://www.bbn.com/NLP/OntoNotes

3. 3 OntoNotes goals  Develop a skeletal representation of the literal meaning of sentences –Add to a frame-based (PropBank) representation of predicates and their arguments: Referring expressions and the textual phrases they refer to Terms disambiguated by coarse-grained word sense in an ontology –Encodes the core, skeletal meaning –Moves away from strings to terms that a reasoning system can use  Find a “sweet spot” in space of –Inter-tagger agreement –Productivity –Depth of representation Text Co-reference Word Sense wrt Ontology Treebank PropBank OntoNotes Annotated Text

4. 4 Creating a Sense Inventory that Supports High Quality Annotation  A large scale annotation effort as part of the OntoNotes project  Two Steps –Grouping subtle, fine-grained WordNet senses into coherent semantic sense groups based on syntactic and semantic criteria For Example: WordNet Sense1: I called my son David WordNet Sense 12: You can call me Sir are grouped together –Annotation

5. 5 Example Grouping: Order –5–5 GroupCharacteristicsExamples 1 “give a command” NP1[+human] ORDER NP2[+animate] to V where NP1 has some authority over NP2 The victim says that the owner ordered the dogs to attack. 2 “request something to be made, supplied, or delivered" NP1[+human] ORDER NP2 I just ordered pizza from panhandle pizza. 3 “organize” I ordered the papers before the meeting,

6. 6 Annotation Process  Verb sense groups are created based on WordNet senses and on- line resources (VerbNet, PropBank, FrameNet, dictionaries)  Newly created verb sense groups are subject to sample- annotation  Verbs higher than 90% ITA (or 85% after regrouping) go to actual annotation  Verbs with less that 90% ITA are regrouped and sent back into sample-annotation tasks.  Regroupings and resample annotations are not done by the original grouper and taggers.  Verbs that complete actual annotation are adjudicated

7. 7 The Grouping and Annotation Process

8. 8 WSD with English OntoNotes Verbs  Picked 217 sense group annotated verbs with 50+ instances each (out of 1300+ verbs) –35K instances total (almost half the data) –WN polysemy range: 59 to 2; Coarse polysemy range: 16 to 2 –Test: 5-fold cross-validation –Automatic performance approaches human performance! WN Avg. Polysemy Onto Avg. Polysemy Baseline ITAMaxEnt SVM 10.45.10.680.8250.8270.822

9. 9 ITA > System Performance

10. 10 ITA < System Performance

11. 11 Discussion  Coarse-grained sense distinctions improve both ITA and system performance  Linguistically motivated features contributed to high system accuracy ALLw/o SEM.w/o SEM+SYN 0.8270.8160.789 Data setBaseline Acc.System Acc.ITA SENSEVAL-2 verbs0.4070.6460.713 OntoNotes verbs0.6800.8270.825

12. 12 Eventive nouns  ISI sense tags nouns  Some nouns have eventive senses –party –development  Given a list of nouns and tagged instances, we nombank just those. A few thousand at most.  Last meeting we reported very poor ITA with Adam’s nombank annotation

13. 13 Comparison of NomBank and PropBank Frames  107 Frames examined: the ISI eventive nouns that have frame files in NomBank  47 of those showed differences between the NomBank and the PropBank frames. Types of differences: 1.No PropBank equivalent 2.No PropBank equivalent for some NomBank senses 3.No NomBank equivalent for some PropBank senses 4.NomBank equivalent has extra Args

14. 14 No PropBank equivalent  15 cases breakdown; downturn; illness; oath; outcome; pain; repercussion; stress; transition; turmoil; unrest

15. 15 No related PropBank equivalent for some NomBank senses  7 cases  No PB equivalent start start.02 “attribute/housing-starts”  PB equivalent of unrelated name appointment appointment.02 “have a date” Equivalent: meet.03, no equivalent sense of “to appoint” PB equivalent of related name has different sense numbering plea plea.02 “beg” Equivalent: plead.01, source is listed as appeal.02

16. 16 solution solution.02 “mix, combine”; source mix.01 Arg0: agent, mixer Arg1: ingredient one Arg2: ingredient two Related PB equivalent: dissolve.01 “cause to come apart”?? Arg0: causer, agent Arg1: thing dissolving Arg2: medium “salt water solution” “rubber solution” “chemical solution”

17. 17 No related NomBank equivalent for some PropBank rolesets  10 cases harassment harassment.01 “bother” Source: harass.01 “bother” “Police and soldiers continue to harass Americans.” “the harassment of diplomats and their families” harass.02 “cause an action” “John harassed Mary into giving him some ice cream.” No NB equivalent.

18. 18 NomBank equivalent has extra Args  6 cases allegation PB: allege.01 Arg0: speaker, alleger Arg1: utterance, allegation Arg2: hearer NB: allegation.01 Arg3: person against whom something is alleged “Fraud allegation against Wei-Chyung Wang” “Abuse alleged against accused murderer. Conclusion: Consider adding Arg3 to VN frame.

19. 19 answer PB: answer.01 Arg0: replier Arg1: in response to Arg2: answer NB: answer.01 Arg3: asker/recipient of answer “Wang’s marketing department provided the sales force Arg3 answers to [the] questions” In PropBank, this role is often fulfilled by the Arg1. “’I’ve read Balzac’, he answers critics Arg1 ”

20. 20 attachment attachment.01 Arg0: agent Arg1: theme Arg2: theme2 Arg3: instrument attach.01 Arg0: agent Arg1: thing being tied Arg2: instrument Attach.01 allows two Arg1’s: John attached the apology note Arg1 to his dissertation Arg1.

21. 21 Other issues  Roles described differently (different label or true difference) utterance.01: Arg0 “agent” utter.01: Arg0 “speaker” score.01 (VN): Arg2 “opponent” score.01 (NB): Arg2 “test/game” “scored against the B team Arg2 ”; “high test Arg2 scores”  Different frame numbers “attach, as with glue” bond.02 (NB) bond.01 (VN)

22. 22 Conclusion  We can fix these!