WORD SENSE DISAMBIGUATION Presented By Roshan R. Karwa Guided By Dr. M.B.Chandak A Technical Seminar on

M OTIVATION One of the open problem in NLP. Computationally determining which sense of a word is activated by its use in a particular context. E.g. I went to the bank to withdraw some money. Needed in: Machine Translation : For correct lexical choice. Information Retrieval: Resolving ambiguity in queries. Information Extraction: For accurate analysis of text. 2

OUTLINE Knowledge Based Approaches WSD using Selectional Preferences Overlap Based Approaches Machine Learning Based Approaches Supervised Approaches Unsupervised Algorithms Hybrid Approaches Summary Conclusion Future Work 3

Knowledge Based Approaches WSD USING SELECTIONAL PREFERENCES AND ARGUMENTs 4 This airlines serves dinner in the evening flight. serve (Verb) agent object – edible This airlines serves the sector between Agra & Delhi. serve (Verb) agent object – sector SENSE 1SENSE 2 o Requires exhaustive enumeration of:  Argument-structure of verbs.  Description of properties of words such that meeting the selectional preference criteria can be decided. E.g. This flight serves the “region” between Mumbai and Delhi

Knowledge Based Approaches: O VERLAP B ASED A PPROACHES Require a Machine Readable Dictionary (MRD). Find the overlap between the features of different senses of an ambiguous word (sense bag) and the features of the words in its context (context bag). The sense which has the maximum overlap is selected as the contextually appropriate sense. 5

OVERLAP BASED APPROACHES: LESK’S A LGORITHM 6 SENSE 1.. THE RESIDUE THAT REMAINS WHEN SOMETHING IS BURNED. SENSE 2.. TIMBER TREE.. SENSE 3.. STRONG ELASTIC WOOD OF ANY OF VARIOUS ASH TREES; USED FOR FURNITURE... ASH “On burning coal we get ash.” In this case Sense 1 of ash would be the winner sense. “ On burning the ash, We found that its root were deap into ground. Winner sense will be again 1 by simple LESK. But Intented meaning is ??????

OVERLAP BASED APPROACHES: WALKER’S ALGORITHM A Thesaurus Based approach. Step 1: For each sense of the target word find the thesaurus category to which that sense belongs. Step 2: Calculate the score for each sense by using the context words. A context words will add 1 to the score of the sense if the thesaurus category of the word matches that of the sense. E.g. The money in this bank fetches an interest of 8% per annum Target word: bank Clue words from the context: money, interest, annum, fetch Sense1: FinanceSense2: Location Money+10 Interest+10 Fetch00 Annum+10 Total30 Context words add 1 to the sense when the topic of the word matches that of the sense

WSD U SING C ONCEPTUAL D ENSITY Select a sense based on the relatedness of that word-sense to the context. Relatedness is measured in terms of conceptual distance (i.e. how close the concept represented by the word and the concept represented by its context words are) This approach uses a structured hierarchical semantic net (WordNet) for finding the conceptual distance. Smaller the conceptual distance higher will be the conceptual density. (i.e. if all words in the context are strong indicators of a particular concept then that concept will have a higher density.) 8

THE JURY(2) PRAISED THE ADMINISTRATION(3) AND OPERATION (8) OF ATLANTA POLICE DEPARTMENT(1 ) Step 1:Lattice Making Step 2: Compute CD Step 3: Select highest CD Step 4: Select concept operation division administrative unit jury committee police department local department government department department juryadministration body CD = CD = C ONCEPTUAL D ENSITY (E XAMPLE )

Supervised Approach: NAÏVE BAYES 10 sˆ= argmax s ε senses Pr(s|V w ) ‘ V w ’ is a feature vector consisting of: POS of w Semantic & Syntactic features of w Collocation vector (set of words around it)  typically consists of next word(+1), next-to-next word(+2), -2, -1 & their POS's Co-occurrence vector (number of times w occurs in bag of words around it) Applying Bayes rule and naive independence assumption sˆ= argmax s ε senses Pr(s). Π i=1 n Pr(V w i |s)

NAÏVE BAYES Example I went to the bank to withdraw some money Collocation vector: Co-occurrence vector: considering window, 2 words before bank and 2 words after bank. So bank appear one time. Vsense of bank:

NAÏVE BAYES Example (CONT..) P(Vbank|sense of bank) = ? P( |sense of bank) = P(N|sense of bank). P(org|sense of bank). P(plural-s|sense of bank). P(went|sense of bank). P(withdraw|sense of bank). P(money|sense of bank) Say, P(org| sense1 bank) P(org| sense2 bank)

E STIMATING P ARAMETERS Parameters in the probabilistic WSD are: Pr(s) Pr(V w i |s) Senses are marked with respect to sense repository (WORDNET) Pr(s)= count(s,w) / count(w) Pr(V w i |s)= Pr(V w i,s)/Pr(s) = c(V w i,s,w)/c(s,w)

Supervised Approach: DECISION LIST ALGORITHM Based on ‘One sense per collocation’ property. Nearby words provide strong and consistent clues as to the sense of a target word. Collect a large set of collocations for the ambiguous word. Calculate word-sense probability distributions for all such collocations. Calculate the log-likelihood ratio Higher log-likelihood = more predictive evidence Collocations are ordered in a decision list, with most predictive collocations ranked highest. 14 Pr(Sense-A| Collocation i ) Pr(Sense-B| Collocation i ) Log( ) 14 Assuming there are only two senses for the word. Of course, this can easily be extended to ‘k’ senses.

Training DataResultant Decision List D ECISION L IST A LGORITHM (C ONTD.) Classification of a test sentence is based on the highest ranking collocation found in the test sentence. E.g. … plucking flowers affects plant growth … 15

Unsupervised Approach: H YPERLEX KEY IDEA Instead of using “dictionary defined senses” extract the “senses from the corpus” itself These “corpus senses” or “uses” correspond to clusters of similar contexts for a word. (river) (water) (flow) (electricity) (victory) (team) (cup) (world) Example: “ Outre la production d electrite, le BARRAGE permettre de-regular le corpav du fleuve” ( In addition to production of electricity, the dam will regulate the river flow)

HYBRID: AN ITERATIVE APPROACH TO WSD Extracts collocational and contextual information form WordNet (gloss) and a small amount of tagged data. Monosemic words in the context serve as a seed set of disambiguated words. It would be interesting to exploit other semantic relations available in WordNet. Combine information obtained from multiple knowledge source

STRUCTURAL SEMANTIC INTERCONNECTIONS (SSI) AN ITERATIVE APPROACH. Uses the following relations hypernymy (car#1 is a kind of vehicle#1) denoted by (kind-of ) hyponymy (the inverse of hypernymy) denoted by (has-kind) meronymy (room#1 has-part wall#1) denoted by (has-part ) holonymy (the inverse of meronymy) denoted by (part-of ) attribute (dry#1 value-of wetness#1) denoted by (attr) gloss denoted by (gloss) context denoted by (context) domain denoted by (dl) Monosemic words serve as the seed set for disambiguation.

Structural Semantic Interconnections (SSI) contd. A SEMANTIC RELATIONS GRAPH FOR THE TWO SENSES OF THE WORD BUS (I.E. VEHICLE AND CONNECTOR)

RESOURCES FOR WSD Sense Inventory:  Dictionaries  Thesauri(roget’s thesaurus.. )  Lexical KB(Wordnet.. ) Corpora:  Raw(Brown corpus.. )  Sense Tagged(Semcor.. )  Automatically Tagged Corpora(Open dictionary Project..)

Conclusion Using a diverse set of features improves WSD accuracy. WSD results are better when the degree of polysemy is reduced. Hyperlex (unsupervised corpus based), SSI (hybrid) look promising for resource-poor Indian languages.

FUTURE IMPLEMENTATION Use unsupervised or hybrid approaches to develop a WSD engine. (focusing on MT) Automatically generate sense tagged data. Explore whether it possible to evaluate the role of WSD in MT.

R EFERENCES Agirre, Eneko & German Rigau "Word sense disambiguation using conceptual density", in Proceedings of the 16th International Conference on Computational Linguistics (COLING), Copenhagen, Denmark, 1996 Ng, Hwee T. & Hian B. Lee "Integrating multiple knowledge sources to disambiguate word senses: An exemplar-based approach", Proceedings of the 34th Annual Meeting of the Association for Computational Linguistics (ACL), Santa Cruz, U.S.A., Ng, Hwee T "Exemplar-based word sense disambiguation: Some recent improvements", Proceedings of the 2nd Conference on Empirical Methods in Natural Language Processing (EMNLP), Providence, U.S.A.,

REFERENCES Rada Mihalcea and Dan Moldovan, 2000."An Iterative Approach to Word Sense Disambiguation", in Proceedings of Florida Artificial Intelligence Research Society Conference (FLAIRS 2000), [pg ] Orlando, FL, May Roberto Navigli, Paolo Velardi, 2005."Structural Semantic Interconnections: A Knowledge-Based Approach to Word Sense Disambiguation", IEEE Transactions On Pattern Analysis and Machine Intelligence, July Yee Seng Chan, Hwee Tou Ng and David Chiang, 2007."Word Sense Disambiguation Improves Statistical Machine Translation", in Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics, Prague, Ping Chen and Chris Bowes, University of Houston-Downtown and Wei Ding and Max Choly, University of Massachusetts, Boston Word Sense Disambiguation with Automatically Acquired Knowledge, 2012 IEEE INTELLIGENT SYSTEMS published by the IEEE Computer Society.

THANK YOU!

APPENDIX Conceptual Density CD(c,m)= m-1 h-1 ∑ nhyp ∕ ∑ nhyp i=o i=0

Unsupervised: LIN’S APPROACH 27 INSTALLATION PROFICIENCY ADEPTNESS READINESS TOILET/BATHROOM WordFreqLog Likelihood ORG Plant Company Industry914.6 Unit99.32 Aerospace25.81 Memory device15.79 Pilot25.37 SENSES OF FACILITYSUBJECTS OF “EMPLOY” Two different words are likely to have similar meanings if they occur in identical local contexts. E.g. The facility will employ 500 new employees. In this case Sense 1 of installation would be the winner sense.