1. Information Retrieval Thesauruses and Cluster Analysis 1

2. Lexicon and Thesaurus Lexicon contains information about words, their morphological variants, and their grammatical usage. Thesaurus relates words by meaning: ship, vessel, sail; craft, navy, marine, fleet, flotilla book, writing, work, volume, tome, tract, codex search, discovery, detection, find, revelation (From Roget's Thesaurus, 1911)

3. Thesaurus in Information Retrieval Use of a thesaurus in indexing (precoordination) A. Manual Used to guide human indexer to assign standard terms and associations. computer-aided instruction see also education UF teaching machines BT educational computing TT computer applications RT education RT teaching From: INSPEC Thesaurus

4. Thesaurus in Information Retrieval Use of a thesaurus in indexing (precoordination) B. Automatic Divide terms into thesaurus classes. Replace similar terms by a thesaurus class. 408dislocation409blast-cooled junctionheat-flow minority-carrierheat-transfer n-p-n p-n-p410 anneal point-contactstrain recombine transition unijunction From: Salton and McGill

5. Desirable Properties for Information Retrieval Thesaurus is specific to a subject area. Contains only terms of interest for identification within that subject area. Ambiguous terms are coded only for the senses important for that field. Target is that each thesaurus class should include terms of moderate frequency. Ideally the classes should have similar frequency.

6. Alexandria Thesaurus: Example canals A feature type category for places such as the Erie Canal. Used for: The category canals is used instead of any of the following. canal bends canalized streams ditch mouths ditches drainage canals drainage ditches... more... Broader Terms: Canals is a sub-type of hydrographic structures.

7. Alexandria Thesaurus: Example (continued) canals (continued) Related Terms: The following is a list of other categories related to canals (non- hierarchial relationships). channels locks transportation features tunnels Scope Note: Manmade waterway used by watercraft or for drainage, irrigation, mining, or water power. » Definition of canals.

8. Art and Architecture Thesaurus Controlled vocabulary for describing and retrieving information: fine art, architecture, decorative art, and material culture. Almost 120,000 terms for objects, textual materials, images, architecture and culture from all periods and all cultures. Used by archives, museums, and libraries to describe items in their collections. Used to search for materials. Used by computer programs, for information retrieval, and natural language processing. A project of the J. Paul Getty Trust

9. Art and Architecture Thesaurus Provides the terminology for objects, and the vocabulary necessary to describe them, such as style, period, shape, color, construction, or use, and scholarly concepts, such as theories, or criticism. Concept: a cluster of terms, one of which is established as the preferred term, or descriptor. Categories: associated concepts, physical attributes, styles and periods, agents, activities, materials, and objects.

10. Art and Architecture Thesaurus: Sample Record Record ID: 198841 Descriptor: rhyta Note: Refers to vessels from Ancient Greece, eastern Europe, or the Middle East that typically have a closed form with two openings, one at the top for filling and one at the base so that liquid could stream out. They are often in the shape of a horn or an animal's head, and were typically used as a drinking cup or for pouring wine into another vessel. Hierarchy: Containers [TQ].................................

11. Art and Architecture Thesaurus: Sample Record (continued) Terms: rhyta rhyton (alternate, singular) protomai protome rhea rheon rheons Related concepts: stirrup cups sturzbechers drinking vessels ceremonial vessels

12. Automatic Thesaurus Construction Approach Select a subject domain. Choose a corpus of documents that cover the domain. Create vocabulary by extracting terms, normalization, precoordination of phrases, etc. Devise a measure of similarity between terms and thesaurus classes. Cluster terms into thesaurus classes, using complete linkage or other cluster method that generates compact clusters.

13. Normalization of vocabulary Normalization rules map variant forms into base expressions. Typical normalization rules for manual thesaurus construction are: (a)Nouns only, or nouns and noun phrases. (b)Singular nouns only. (c)Spelling (e.g., U.S.). (d)Capitalization, punctuation (e.g., hyphens), initials (e.g., IBM), abbreviations (e.g., Mr.). Usually, many possible decisions can be made, but they should be followed consistently. Which of these can be carried out automatically with reasonable accuracy?

14. Terms to include Only terms that are likely to be of interest for content identification. High-frequency terms should be ignored (large stop-list). Ambiguous terms should be coded for the senses likely to be important in the document collection. Each thesaurus class should have approximately the same frequency of occurrence. Terms of negative discrimination should be eliminated. after Salton and McGill

15. Discriminant value Discriminant value is the degree to which a term is able to discriminate between the documents of a collection = (average document similarity without term k) - (average document similarity with term k) Good discriminators decrease the average document similarity Note that this definition uses the document similarity.

16. Incidence array D 1 : alpha bravo charlie delta echo foxtrot golf D 2 : golf golf golf delta alpha D 3 : bravo charlie bravo echo foxtrot bravo D 4 : foxtrot alpha alpha golf golf delta alphabravocharliedeltaechofoxtrotgolf D 1 1 1 1 1 1 1 1 D 2 1 1 1 D 3 1 1 1 1 D 4 1 1 1 1 73447344

17. Document similarity matrix D 1 D 2 D 3 D 4 D 1 0.65 0.760.76 D 2 0.65 0.000.87 D 3 0.760.00 0.25 D 4 0.760.87 0.25 Average similarity = 0.55

18. Discriminant value Repeat excluding one term at a time withoutaverage similarityDV alpha0.53 -0.02 bravo0.56+0.01 charlie0.56+0.01 delta0.53 -0.02 echo0.56+0.01 foxtrot0.52-0.03 golf0.53-0.02 Average similarity = 0.55 alpha, delta, foxtrot, golf are bad discriminator s

19. Phrase construction In a precoordinated thesaurus, term classes may contain phrases. Informal definitions: pair-frequency (i, j) is the frequency that a given pair of words occur in context (e.g., in succession within a sentence) phrase is a pair of words, i and j that occur in context with a higher frequency than would be expected from their overall frequency cohesion (i, j) = observed pair-frequency (i, j) expected pair-frequency if i, j independent

20. Phrase construction: simple case Example: corpus of n terms p i, j is the observed frequency that a given pair of terms occur in succession. f i is the number of occurrences of term i in the corpus. There are n-1 pairs. If the terms are independent, the probability that a given pair begins with term i and ends with term j is (f i /n).(f j /n) cohesion (i, j) = n 2.p i, j (n-1)f i.f j

21. Phrase construction Salton and McGill algorithm 1. Computer pair-frequency for all terms. 2. Reject all pairs that fall below a certain frequency threshold 3. Calculate cohesion values 4. If cohesion above a threshold value, consider word pair as a phrase. Automatic phrase construction by statistical methods is rarely used in practice. There is promising research on phrase identification using methods of computational linguistics

22. Similarities The vocabulary consists of a set of elements, each of which can be a single term or a phrase. The next step is to calculate a measure of similarity between elements. One measure of similarity is the number of documents that have terms i and k in common: S(t j, t k ) =  t ij t ik where t ij = 1 if document i contains term j and 0 otherwise. i=1 n

23. Similarities: Incidence array alphabravocharliedeltaechofoxtrotgolf D 1 1 1 1 1 1 1 1 D 2 1 1 1 D 3 1 1 1 1 D 4 1 1 1 1 n 3 2 2 3 2 3 3

24. Term similarity matrix alphabravocharliedeltaechofoxtrotgolf alpha 1 1 3 1 2 3 bravo 1 2 1 2 2 1 charlie 1 2 1 2 2 1 delta 3 1 1 1 2 3 echo 1 2 2 1 2 1 foxtrot 2 2 2 2 2 2 golf 3 1 1 3 1 2 Using count of documents that have two terms in common

25. Similarity measures Improved similarity measures can be generated by: Using term frequency matrix instead of incidence matrix Weighting terms by frequency: cosine measure  t ij t ik |t j | |t k | dice measure  t ij t ik  t ik +  t ij i=1 n n n n S(t j, t k ) =

26. Term similarity matrix alphabravocharliedeltaechofoxtrotgolf alpha 0.2 0.2 0.5 0.2 0.33 0.5 bravo 0.2 0.5 0.2 0.5 0.4 0.2 charlie 0.2 0.5 0.2 0.5 0.4 0.2 delta 0.5 0.2 0.2 0.2 0.33 0.5 echo 0.2 0.5 0.5 0.2 0.4 0.2 foxtrot 0.33 0.4 0.4 0.33 0.4 0.33 golf 0.5 0.2 0.2 0.5 0.2 0.33 Using incidence matrix and dice weighting

27. Clustering terms to form concepts The final stage is to group similar terms together into concepts. This is done by cluster analysis. Cluster analysis is the topic of the next lecture.