1. Category-Based Pseudowords
Preslav Nakov & Marti Hearst
University of California at Berkeley
EECS & SIMS
Supported by Genentech and ARDA Aquaint
11/19/2018
HLT/NAACL'03

2. Word sense disambiguation
WSD task: determine the sense of a particular instance of a multi-sense word given its context
classic ambiguous example: bank
homography
river bank
financial institution
polysemy
building
11/19/2018
HLT/NAACL'03

3. Evaluation Ideally: using a sense-tagged corpus Moving to a new domain
general purpose – e.g. SENSEVAL corpus
specific domain, e.g. biomedical
the National Library of Medicine test collection
contains instances of 50 highly frequent ambiguous concepts from the UMLS Metathesaurus.
Moving to a new domain
a sense-tagged corpus may be unavailable
even when available, may be unsuitable
What if we use a different sense distinction: e.g. MeSH instead of the UMLS Metathesaurus?
What if we are also interested in less frequent words, e.g. need to evaluate an all-words system?
11/19/2018
HLT/NAACL'03

4. Pseudowords
building a sense-tagged corpus is very expensive, so create an artificial one
pseudoword: composite comprised of two or more words, chosen at random (Gale et al.’92), (Schuetze’92):
e.g. banana and door  banana_door
accepted as an upper bound of the true system’s accuracy
11/19/2018
HLT/NAACL'03

5. Problems Chosen entirely at random, and thus:
difficult to characterize in terms of the type of ambiguity being modeled
optimistic in their estimations (Gaustad’01)
highly likely to combine semantically distinct words
real ambiguous words have senses similar in meaning and difficult to distinguish
11/19/2018
HLT/NAACL'03

6. Use lexical category membership
The solution
Use lexical category membership
11/19/2018
HLT/NAACL'03

7. MeSH and Medline we use MeSH (Medical Subject Headings)
example: Eye has the following codes A (child of Face)
A (child of Sense Organs)
average number of senses: 2.12
we cut after the first period to allow generalization (e.g. A01 and A09)
71.18% - single class, 22.14% - two classes
the ambiguity drops to 1.39
Medline abstracts - 180,226
training: 120,150
testing: 60,076
11/19/2018
HLT/NAACL'03

8. Pseudowords generation (1)
Build a list C of the category couples and their frequencies in the training corpus
11/19/2018
HLT/NAACL'03

9. Pseudowords generation (2)
Generate pseudowords with the following characteristics:
represent a real ambiguity class pair (met in the training corpus)
the number of pseudowords drawn from a particular class pair is proportional to the pair’s frequency
only unambiguous words are used as pseudowords constituents
multi-word concepts are allowed as elements, e.g.
general systems theory +
glutathione s-tranferase
11/19/2018
HLT/NAACL'03

10. Pseudowords generation (3)
Pseudowords for the lower bound
in real texts, the more frequent sense for a two-sense distinction occurs around 92% of the time (Sanderson & van Rijsbergen’99)
evenly distributed senses are harder
so we build a balanced list W of pairs:
we calculate the mean corpus word frequency E and then find the words with freq. in [E/2;3E/2]
in the particular experiment:
E=45.21, which gave a list of 64,596 pairs
11/19/2018
HLT/NAACL'03

11. Pseudowords generation (4)
importance sampling
1) Select a category pair c1,c2 from C by sampling from a multinomial distribution with parameters proportional to the frequencies of the elements of C.
2) Sample uniformly to draw two random distinct words w1 and w2 whose classes correspond to the classes selected in step 1).
3) If the word pair w1,w2 has been sampled already, go to step 1) and try again.
we sampled 1,000 pseudowords (88,758 instances) out of the possible 64,596
11/19/2018
HLT/NAACL'03

12. Sample pseudowords
the more unusual pairs come from less frequent categories
11/19/2018
HLT/NAACL'03

13. Classifier Naïve Bayes classifier we used a symmetric context window:
simple, commonly used for WSD, and among the best performing
we used a symmetric context window:
10, 20, 40 and 300 words on each side
category name as a proxy for the sense
ambiguous MeSH categories as target
UNambiguous MeSH categories as features (we use a class-based model, and not a word-based one)
11/19/2018
HLT/NAACL'03

14. used an abbr. extraction tool described in (Schwartz&Hearst’03)
Abbreviations
we have no real disambiguated corpus
use abbreviations, as suggested in (Liu et al.,’02)
represent real ambiguous words
but may be due to accident
intermediate position between entirely random pseudowords and real ambiguous words
we generated 98,841 abbreviations (332,020 instances in total) such that:
their expansions are fully and unambiguously mapped to MeSH
they represent exactly two distinct categories
used an abbr. extraction tool described in (Schwartz&Hearst’03)
11/19/2018
HLT/NAACL'03

15. Sample abbreviations
11/19/2018
HLT/NAACL'03

16. Evaluation Category based Non-category based
baseline – choose the more frequent class (shown for abbreviations)
pessimistic – evenly distributed constituents
realistic – random constituents (frequency at least 5)
abbreviations
Non-category based
optimistic – completely random (the standard way to generate)
11/19/2018
HLT/NAACL'03

17. Conclusions
We introduced category based pseudowords based on distributions from lexical category co-occurrence:
give a more accurate lower bound
allow detailed study (many samples) of a particular sense ambiguity
represent a better motivated word grouping in pseudowords
11/19/2018
HLT/NAACL'03

18. Thank you!
Your questions?
11/19/2018
HLT/NAACL'03