Text Mining In InQuery Vasant Kumar, Peter Richards August 25th, 1999.

History InQuery was originally a research product from Center for Intelligent Information Retrieval at the University of Massachusetts, Amherst A commercial-strength InQuery API from Sovereign Hill Software InQuery 5.0 with LCA and Graphical User Interface

Outline Text mining Text mining using “local context analysis” (LCA). Text mining using “top concepts” Concept recognizers Demonstration of LCA and “top concepts” Q & A

Text Mining Helps find needle in the hay stack Query expansion Discovers interesting relationships between concepts Discovers characteristics about the database

Concepts Words Noun phrases People names Company names User-defined concepts

Local Context Analysis (LCA) Associates a query to a ranked list of concepts for several concept types (noun phrases, people names..) Concept association is done on the fly –no complex databases to be created –changes to the database are immediately taken into account.

Background Unit of retrieval is a passage (local context), in contrast to a document in regular search. A passage is a window of words of length n Overlapping passages are used.

LCA Process Generate candidate passages (sub- documents) Extract concepts and their statistics Apply LCA algorithm to rank the concepts for each concept type

Step 1: Generate Candidate Passages The documents are split into passages (virtual sub-documents) Evaluate the query on these passages to generate a weight for each passage Rank the passages Select the top m best passages

Step 2: Extract Concepts Extract the passages from their respective documents for all the passages in the candidate passage list. Each passage in the candidate list is passed through a set of “concept recognizers” to extract respective concept lists. Generate passage level statistics for all concepts and query terms

Step 3: Apply LCA Algorithm Generate local context statistics for concepts and query terms (specific to the set of candidate passages) Use LCA algorithm to generate weights for concepts. The passage level and local context level statistics are used.. Rank the concepts and select top n The above steps are repeated for all concept types.

Text Mining Using Top Concepts Retrieve documents Extract concepts from each document using “concept recognizers” Generate most frequently occurring concepts for all concept types. Persist the most frequently occurring concepts.

Noun Phrase Recognizer Tokenization to generate words Parts-of-speech tagging (noun, verb, etc.) Select noun phrases

Other Recognizers Company and people name recognizers –based on pattern matching rules –uses external lists of names for normalization and additional evidence. User-defined recognizer –uses a user provided list of concepts (single/multiword) –generates a state machine

Demonstration of LCA and Top Concepts in InQuery 5.1