CIS630 1 Penn Different Sense Granularities Martha Palmer, Olga Babko-Malaya September 20, 2004

CIS630 2 Penn Statistical Machine Translation results  CHINESE TEXT  The japanese court before china photo trade huge & lawsuit.  A large amount of the proceedings before the court dismissed workers.  japan’s court, former chinese servant industrial huge disasters lawsuit.  Japanese Court Rejects Former Chinese Slave Workers’ Lawsuit for Huge Compensation.

CIS630 3 Penn Outline  MT example  Sense tagging Issues highlighted by Senseval1  Senseval2  Groupings,  Impact on ITA  Automatic WSD, impact on scores

CIS630 4 Penn WordNet - Princeton  On-line lexical reference (dictionary)  Words organized into synonym sets concepts  Hypernyms (ISA), antonyms, meronyms (PART)  Useful for checking selectional restrictions  (doesn’t tell you what they should be)  Typical top nodes - 5 out of 25  (act, action, activity)  (animal, fauna)  (artifact)  (attribute, property)  (body, corpus)

CIS630 5 Penn WordNet – president, 6 senses 1.president -- (an executive officer of a firm or corporation) -->CORPORATE EXECUTIVE, BUSINESS EXECUTIVE…  LEADER 2. President of the United States, President, Chief Executive -- (the person who holds the office of head of state of the United States government; "the President likes to jog every morning") -->HEAD OF STATE, CHIEF OF STATE 3. president -- (the chief executive of a republic) -->HEAD OF STATE, CHIEF OF STATE 4. president, chairman, chairwoman, chair, chairperson -- (the officer who presides at the meetings of an organization; "address your remarks to the chairperson") --> PRESIDING OFFICER  LEADER 5. president -- (the head administrative officer of a college or university) --> ACADEMIC ADMINISTRATOR ….  LEADER 6. President of the United States, President, Chief Executive -- (the office of the United States head of state; "a President is elected every four years") --> PRESIDENCY, PRESIDENTSHIP  POSITION

CIS630 6 Penn Limitations to WordNet  Poor inter-annotator agreement (73%)  Just sense tags - no representations  Very little mapping to syntax  No predicate argument structure  no selectional restrictions  No generalizations about sense distinctions  No hierarchical entries

CIS630 7 Penn SIGLEX98/SENSEVAL  Workshop on Word Sense Disambiguation  54 attendees, 24 systems, 3 languages  34 Words ( Nouns, Verbs, Adjectives )  Both supervised and unsupervised systems  Training data, Test data  Hector senses - very corpus based (mapping to WordNet)  lexical samples - instances, not running text  Replicability over 90%, ITA 85% ACL-SIGLEX98,SIGLEX99, CHUM00

CIS630 8 Penn Hector - bother, 10 senses  1. intransitive verb, - (make an effort), after negation, usually with to infinitive; (of a person) to take the trouble or effort needed (to do something). Ex. “About 70 percent of the shareholders did not bother to vote at all.”  1.1 (can't be bothered), idiomatic, be unwilling to make the effort needed (to do something), Ex. ``The calculations needed are so tedious that theorists cannot be bothered to do them.''  2. vi; after neg; with `about" or `with"; rarely cont – (of a person) to concern oneself (about something or someone) “He did not bother about the noise of the typewriter because Danny could not hear it above the sound of the tractor.”  2.1 v-passive; with `about" or `with“ - (of a person) to be concerned about or interested in (something) “The only thing I'm bothered about is the well-being of the club.”

CIS630 9 Penn Mismatches between lexicons: Hector - WordNet, shake

CIS Penn VERBNET

CIS Penn VerbNet/WordNet

CIS Penn Mapping WN-Hector via VerbNet SIGLEX99, LREC00

CIS Penn SENSEVAL2 –ACL’01 Adam Kilgarriff, Phil Edmond and Martha Palmer All-words taskLexical sample task CzechBasque DutchChineseEnglish EstonianItalian Japanese Korean Spanish Swedish

CIS Penn English Lexical Sample - Verbs  Preparation for Senseval 2  manual tagging of 29 highly polysemous verbs (call, draw, drift, carry, find, keep, turn,...)  WordNet (pre-release version 1.7)  To handle unclear sense distinctions  detect and eliminate redundant senses  detect and cluster closely related senses NOT ALLOWED

CIS Penn WordNet – call, 28 senses 1.name, call -- (assign a specified, proper name to; "They named their son David"; "The new school was named after the famous Civil Rights leader") -> LABEL 2. call, telephone, call up, phone, ring -- (get or try to get into communication (with someone) by telephone; "I tried to call you all night"; "Take two aspirin and call me in the morning") ->TELECOMMUNICATE 3. call -- (ascribe a quality to or give a name of a common noun that reflects a quality; "He called me a bastard"; "She called her children lazy and ungrateful") -> LABEL

CIS Penn WordNet – call, 28 senses 4. call, send for -- (order, request, or command to come; "She was called into the director's office"; "Call the police!") -> ORDER 5. shout, shout out, cry, call, yell, scream, holler, hollo, squall -- (utter a sudden loud cry; "she cried with pain when the doctor inserted the needle"; "I yelled to her from the window but she couldn't hear me") -> UTTER 6. visit, call in, call -- (pay a brief visit; "The mayor likes to call on some of the prominent citizens") -> MEET

CIS Penn Groupings Methodology  Double blind groupings, adjudication  Syntactic Criteria (VerbNet was useful)  Distinct subcategorization frames  call him a bastard  call him a taxi  Recognizable alternations – regular sense extensions:  play an instrument  play a song  play a melody on an instrument

CIS Penn Groupings Methodology (cont.)  Semantic Criteria  Differences in semantic classes of arguments  Abstract/concrete, human/animal, animate/inanimate, different instrument types,…  Differences in entailments  Change of prior entity or creation of a new entity?  Differences in types of events  Abstract/concrete/mental/emotional/….  Specialized subject domains

CIS Penn WordNet: - call, 28 senses WN2, WN13,WN28 WN15 WN26 WN3 WN19 WN4 WN 7 WN8 WN9 WN1 WN22 WN20 WN25 WN18 WN27 WN5 WN 16WN6 WN23 WN12 WN17, WN 11 WN10, WN14, WN21, WN24

CIS Penn WordNet: - call, 28 senses, groups WN2, WN13,WN28 WN15 WN26 WN3 WN19 WN4 WN 7 WN8 WN9 WN1 WN22 WN20 WN25 WN18 WN27 WN5 WN 16WN6 WN23 WN12 WN17, WN 11 WN10, WN14, WN21, WN24, Phone/radio Label Loud cry Bird or animal cry Request Call a loan/bond Visit Challenge Bid

CIS Penn WordNet – call, 28 senses, Group1 1.name, call -- (assign a specified, proper name to; "They named their son David"; "The new school was named after the famous Civil Rights leader") --> LABEL 3. call -- (ascribe a quality to or give a name of a common noun that reflects a quality; "He called me a bastard"; "She called her children lazy and ungrateful") --> LABEL 19. call -- (consider or regard as being; "I would not call her beautiful")--> SEE 22. address, call -- (greet, as with a prescribed form, title, or name; "He always addresses me with `Sir'"; "Call me Mister"; "She calls him by first name") --> ADDRESS

CIS Penn Sense Groups: verb ‘develop’ WN1 WN2 WN3 WN4 WN6 WN7 WN8 WN5 WN 9 WN10 WN11 WN12 WN13 WN 14 WN19 WN20

CIS Penn Groups 1 and 2 of Develop GroupSense No. GlossHypernym 1 – Abstract WN1 WN2 Products, or mental creations Mental creations – “new theory” Gradually unfold – “the plot …” create 2 – New (property) WN3 WN4 Personal attribute – “a passion for …”Physical characteristic – “a beard” change

CIS Penn Group 3 of Develop GroupSense No. GlossHypernym 3 – New (self) WN5 WN9 WN10 WN14 WN20 Originate- “new religious movement” Gradually unfold – “the plot …” Grow – “a flower developed …” Mature – “The child developed …” Happen – “report the news as it …” become occur grow change occur

CIS Penn Group 4 of Develop GroupSense No. GlossHypernym 4 – Improve item WN6 WN7 WN8 WN11 WN12 WN13 WN19 Resources – “natural resources” Ideas – “ideas in your thesis” Train animate beings – “violinists” Civilize – “developing countries” Make, grow – “develop the grain” Business – “develop the market” Music – “develop the melody” improve theorize teach change generate complicate

CIS Penn Maximum Entropy WSD Hoa Dang (in progress)  Maximum entropy framework  combines different features with no assumption of independence  estimates conditional probability that W has sense X in context Y, (where Y is a conjunction of linguistic features  feature weights are determined from training data  weights produce a maximum entropy probability distribution

CIS Penn Features used  Topical contextual linguistic feature for W:  presence of automatically determined keywords in S  Local contextual linguistic features for W:  presence of subject, complements  words in subject, complement positions, particles, preps  noun synonyms and hypernyms for subjects, complements  named entity tag (PERSON, LOCATION,..) for proper Ns  words within +/- 2 word window

CIS Penn Maximum Entropy WSD Hoa Dang, Senseval2 Verbs (best)  Maximum entropy framework, p(sense|context)  Contextual Linguistic Features  Topical feature for W: +2.5%,  keywords (determined automatically)  Local syntactic features for W: +1.5 to +5%,  presence of subject, complements, passive?  words in subject, complement positions, particles, preps, etc.  Local semantic features for W: +6%  Semantic class info from WordNet (synsets, etc.)  Named Entity tag (PERSON, LOCATION,..) for proper Ns  words within +/- 2 word window

CIS Penn Results - first 5 Senseval2 verbs VerbBeginCallCarryDevelopDrawDress WN/corpus 10/9 28/14 39/2221/1635/2115/8 Grp/corp 10/911/716/119/615/97/4 Entropy ITA-fine ITA-coarse MX-fine MX-coarse

CIS Penn Results – averaged over 28 verbs Total WN/corpus 16.28/10.83 Grp/corp 8.07/5.90 Entropy 2.81 ITA-fine 71% ITA-coarse 82% MX-fine 59% MX-coarse 69%

CIS Penn Grouping improved sense identification for MxWSD  75% with training and testing on grouped senses vs. 43% with training and testing on fine-grained senses  Most commonly confused senses suggest grouping:  (1) name, call--assign a specified proper name to; ``They called their son David''  (2) call--ascribe a quality to or give a name that reflects a quality; ``He called me a bastard'';  (3) call--consider or regard as being; ``I would not call her beautiful''  (4) address, call--greet, as with a prescribed form, title, or name; ``Call me Mister''; ``She calls him by his first name''

CIS Penn Criteria to split Framesets  Semantic classes of arguments, such as animacy vs. inanimacy Serve 01. Act, work  Group 1: function (His freedom served him well)  Group 2: work (He served in Congress)

CIS Penn Criteria to split Framesets  Semantic type of event (abstract vs. concrete) See 01. View  Group 1: Perceive by sight (Can you see the bird?)  Group 5: determine, check (See whether it works)

CIS Penn Overlap with PropBank Framesets WN5, WN16,WN12 WN15 WN26 WN3 WN19 WN4 WN 7 WN8 WN9 WN1 WN22 WN20 WN25 WN18 WN27 WN2 WN 13WN6 WN23 WN28 WN17, WN 11 WN10, WN14, WN21, WN24, Loud cry Label Phone/radio Bird or animal cry Request Call a loan/bond Visit Challenge Bid

CIS Penn Overlap between Senseval2 Groups and Framesets – 95% WN1 WN2 WN3 WN4 WN6 WN7 WN8 WN5 WN 9 WN10 WN11 WN12 WN13 WN 14 WN19 WN20 Frameset1 Frameset2 develop

CIS Penn Framesets →Groups→ WordNet WN1 WN2 WN9 WN8 WN3 WN4 WN12 WN5 WN16 WN18 WN14 WN7 WN15 WN10 WN6 WN13 Frameset1 Frameset2 drop WN11 Frameset3

CIS Penn Groups 1 and 2 of Develop GroupSense No. GlossHypernym 1 – Abstract WN1 WN2 Products, or mental creations Mental creations – “new theory” Gradually unfold – “the plot …” create 2 – New (property) WN3 WN4 Personal attribute – “a passion for …”Physical characteristic – “a beard” change

CIS Penn Group 3 of Develop GroupSense No. GlossHypernym 3 – New (self) WN5 WN9 WN10 WN14 WN20 Originate- “new religious movement” Gradually unfold – “the plot …” Grow – “a flower developed …” Mature – “The child developed …” Happen – “report the news as it …” become occur grow change occur

CIS Penn Translations of Develop groups GroupSense No.PortugueseGerman G4 G1 G2 G4 G3 WN13 markets WN1 products WN2 ways WN2 theory WN3 understanding WN2 character WN10 bacteria WN5 movements desenvolver desenvolver-se entwickeln bilden ausbilden bilden sich bilden

CIS Penn Translations of Develop groups GroupSense No.ChineseKorean G4 G1 G2 G4 G3 WN13 markets WN1 products WN2 ways WN2 theory WN3 understanding WN2 character WN10 bacteria WN5 movements kai1-fa1 fa1-zhan3 pei2-yang3-chu1 pei2-yang3 fa1-yu4 xing2-cheng2 hyengsengha-ta kaypalha-ta palcensikhi-ta yangsengha-ta paltalha-ta hyengsengtoy-ta

CIS Penn An Example of Mapping: verb ‘serve’ Assignment: Do you agree? Frameset id = serve.01 Sense Groups serve 01: Act, work Roles: Arg0:worker Arg1:job, project Arg2:employer GROUP 1: WN1 (function) WN3(contribute to) WN12 (answer) GROUP 2: WN2 (do duty) WN13 (do military service) GROUP 3: WN4 (be used by) WN8 (serve well) WN14 (service) GROUP 5: WN7 (devote one’s efforts) WN10 (attend to)

CIS Penn Frameset Tagging Results: overall accuracy 90%* (baseline 73.5%) VerbFramesetsInstancesAccuracy call carry develop draw leave pull serve use work * Gold Standard parses

CIS Penn Sense Hierarchy  PropBank Framesets – ITA 94% coarse grained distinctions 20 Senseval2 verbs w/ > 1 Frameset Maxent WSD system, 73.5% baseline, 90% accuracy  Sense Groups (Senseval-2) - ITA 82% (now 89%) Intermediate level (includes Levin classes) – 69%  WordNet – ITA 71% fine grained distinctions, 60.2%

CIS Penn Summary of WSD  Choice of features is more important than choice of machine learning algorithm  Importance of syntactic structure (English WSD but not Chinese)  Importance of dependencies  Importance of an hierarchical approach to sense distinctions, and quick adaptation to new usages.