1 Chapter 7 Text Operations

2 Logical View of a Document document structure recognition text+ structure accents, spacing, etc. stopwords noun groups stemming automatic or manual indexing structure text full text index terms

3 Text Operations Lexical analysis of the text Elimination of stopwords Stemming Selection of index terms Construction of term categorization structures

4 Lexical Analysis of the Text Word separators  space  digits  hyphens  punctuation marks  the case of the letters

5 Lexical Analysis for Automatic Indexing Lexical Analysis Convert an input stream of characters into stream words or token. What is a word or a token? Tokens consist of letters.  digits: Most numbers are not good index terms. counterexamples: case numbers in a legal database, “ B6 ” and “ B12 ” in vitamin database.

6 Lexical Analysis for Automatic Indexing (Continued)  hyphens break hyphenated words: state-of-the- art, state of the art keep hyphenated words as a token: “ Jean-Claude ”, “ F-16 ”  punctuation marks: often used as parts of terms, e.g., OS/2, 510B.C.  case: usually not significant in index terms

7 Elimination of Stopwords A list of stopwords  words that are too frequent among the documents  articles, prepositions, conjunctions, etc. Can reduce the size of the indexing structure considerably Problem  Search for “ to be or not to be ” ?

8 Stopword Avoid retrieving almost very item in a database regardless of its relevance. Examples  conservative approach (ORBIT Search Service): and, an, by, from, of, the, with  (derived from Brown corpus): 425 words a, about, above, across, after, again, against, all, almost, alone, along, already, also, although, always, among, an, and, another, any, anybody, anyone, anything, anywhere, are, area, areas, around, as, ask, asked, asking, asks, at, away, b, back, backed, backing, backs, be, because, became,... Articles, prepositions, conjunctions, …

9 Stemming Example  connect, connected, connecting, connection, connections  effectiveness --> effective --> effect  picnicking --> picnic Removing strategies  affix removal: intuitive, simple  table lookup  successor variety  n-gram

10 Stemmers programs that relate morphologically similar indexing and search terms stem at indexing time  advantage: efficiency and index file compression  disadvantage: information about the full terms is lost example (CATALOG system), stem at search time Look for: system users Search Term: users TermOccurrences 1. user15 2. users1 3. used3 4. using2 The user selects the terms he wants by numbers

11 Conflation Methods manual automatic (stemmers)  affix removal longest match vs. simple removal  successor variety  table lookup  n-gram evaluation  correctness  retrieval effectiveness  compression performance TermStem engineeringengineer engineeredengineerengineer

12 Successor Variety Definition the number of different characters that follow it in words in some body of text Example a body of text: able, axle, accident, ape, about successor variety of apple 1st: 4 (b, x, c, p) 2nd: 1 (e)

13 Successor Variety (Continued) Idea The successor variety of substrings of a term will decrease as more characters are added until a segment boundary is reached, i.e., the successor variety will sharply increase. Example Test word: READABLE Corpus: ABLE, BEATABLE, FIXABLE, READS, READABLE, READING, RED, ROPE, RIPE PrefixSuccessor VarietyLetters R3E, O, I RE2A, D REA 1D READ3A, I, S READA1B READAB1L READABL1E READABLE 1blank

14 n-gram stemmers diagram a pair of consecutive letters shared diagram method association measures are calculated between pairs of terms where A: the number of unique diagrams in the first word, B: the number of unique diagrams in the second, C: the number of unique diagrams shared by A and B.

15 n-gram stemmers (Continued) Example statistics => st ta at ti is st ti ic cs unique diagrams => at cs ic is st ta ti statistical => st ta at ti is st ti ic ca al unique diagrams => al at ca ic is st ta ti

16 n-gram stemmers (Continued) similarity matrix determine the semantic measures for all pairs of terms in the database word 1 word 2 word 3...word n-1 word 1 wrod 2 S 21 word 3 S 31 S 32.. word n S n1 S n2 S n3 … S n(n-1) terms are clustered using a single link clustering method more a term clustering procedure than a stemming one

17 Q-gram stemmers Example Define Q=3 (L+Q-1) statistics => =12 { ##s,#st,sta,tat,ati,tis,ist,sti,tic,ics,cs#,s## } statistical => =13 {##s,#st,sta,tat,ati,tis,ist,sti,tic,ica,cal,al#,l##} Q-gram Distance = 4

18 Affix Removal Stemmers procedure Remove suffixes and/or prefixes from terms leaving a stem, and transform the resultant stem. example : plural forms If a word ends in “ ies ” but not “ eies ” or “ aies ” then “ ies ” --> “ y ” If a word ends in “ es ” but not “ aes ”, “ ees ”, or “ oes ” then “ es ” --> “ e ” If a word ends in “ s ”, but not “ us ” or “ ss ” then “ s ” --> NULL

19 Index Terms Selection Motivation  A sentence is usually composed of nouns, pronouns, articles, verbs, adjectives, adverbs, and connectives.  Most of the semantics is carried by the noun words. Identification of noun groups  A noun group is a set of nouns whose syntactic distance in the text does not exceed a predefined threshold

20 Index Terms Selection Indexing by single words  single words are often ambiguous and not specific enough for accurate discrimination of documents  bank terminology vs. terminology bank Indexing by phrases  Syntactic phrases are almost always more specific than single words Indexing by single words and phrases

21 Thesauri Peter Roget, 1988 Example cowardly adj. Ignobly lacking in courage: cowardly turncoats Syns: chicken (slang), chicken-hearted, craven, dastardly, faint-hearted, gutless, lily-livered, pusillanimous, unmanly, yellow (slang), yellow-bellied (slang). A controlled vocabulary for the indexing and searching

22 The Purpose of a Thesaurus To provide a standard vocabulary for indexing and searching To assist users with locating terms for proper query formulation To provide classified hierarchies that allow the broadening and narrowing of the current query request

23 Functions of thesauri Provide a standard vocabulary for indexing and searching Assist users with locating terms for proper query formulation Provide classified hierarchies that allow the broadening and narrowing of the current query request

24 Usage Indexing Select the most appropriate thesaurus entries for representing the document. Searching Design the most appropriate search strategy.  If the search does not retrieve enough documents, the thesaurus can be used to expand the query.  If the search retrieves too many items, the thesaurus can suggest more specific search vocabulary.

25 Document Clustering Global clustering  The grouping of documents accordingly to their occurrence in the whole collection Local clustering:  The grouping of the local set of retrieved documents by a query

26 Hypercentroid Supercentroids Centroids Documents Typical Clustered File Organization complete space superclusters clusters

27 Search Strategy for Clustered Documents CentroidsDocuments Typical Search path Highest-level centroid Supercentroids Centroids Documents

28 Text Compression Finding ways to represent the text in fewer bits or bytes Statistical Methods  Huffman coding Dictionary-based  Ziv-Lempel Inverted File compression