Information Retrieval on the World Wide Web Authors: Venkat N. Gudivada Vijay V. Raghavan William I. Grosky Rajesh Kasanagottu Presented by Rob von Behren

Roadmap Information Retrieval Implementation Issues and Techniques Analysis

Definitions Information retrieval - querying against a set of documents to find a subset of "relevant" documents Objective terms - external descriptions, not related to content Nonobjective terms (content terms) - descriptions of the informational content of the document indexing exhaustivity - degree to which the index covers the document space term specificity - describes how well a particular term limits search results. Recall - relevant docs found / relevant docs in collection Precision - relevant docs found / total docs found

Key qualities Document and query representations Mechanisms for finding relevant documents and ranking the results Mechanisms for obtaining user feedback

Types of IR models Set Theoretic Algebraic Probabilistic Hybrid

Set Theoretic Models Boolean model - Simple Boolean queries regarding existence of terms within documents. Queries do not contain information about the context of the terms. fuzzy set model - Slight expansion of Boolean. Allows results to include documents that meet most of the requirements of the Boolean search.

Algebraic models (vector-space model) Documents are represented by n-dimensional vectors. Typically one dimension per term Also possible to treat signatures as bit vectors Queries are n-dimensional vectors Query relevance is the scalar product of the document with the query

Probabilistic models Start with some user-supplied relevance information about a “training set” of documents Compute P(relevant | T) and P(non-relevant |T) based on the terms observed in the training set Useful for theoretical analysis, but probably not in practice (?)

Hybrid models (extended Boolean model) Represent documents as vectors Use the L-p norm, to allow definition of Boolean operations on vectors p=1 ==> the vector model p=infinity & terms are equally weighted ==> Boolean model Empirically best values: 2 <= p <= 5

User feedback Modify query representation (can be done by the user) modify term weights query expansion (add new terms) split the query Modify document representation change term weights within the database agent-based filtering

Roadmap Information Retrieval Implementation Issues and Techniques Analysis

Web Crawling WWW is a directed graph starting points: Use your favorite graph traversal algorithm!! Netizenship issues starting points: individual page set of pages domain name searching (good because the web isn't necessarily connected)

Automatic Indexing single term - Just look at the existence or non-existence of the term in the document phrase - Additionally store other information about the position of the term in the document, and the positions of other terms relative to it

Automatic Indexing (cont) Statistical - Term weights depend on how well they differentiate between documents Information-theoretic - Signal to noise. Similar to some types of statistical indexing Probabilistic - Compute the importance of terms based on user feedback on a subset of the documents linguistic - Use language syntax information such as part of speech

Current Search Engines type 1: automatically indexed type 2: (partially) human indexed, hierarchically organized Common features allow Boolean searches do vector-like queries to find document relevance

Current Search Engines (cont) Type 1 AltaVista, Excite, HotBot, InfoSeek, Lycos, OpenText Type 2 Yahoo, Magellan, WWW Virtual Library, Galaxy

Roadmap Information Retrieval Implementation Issues and Techniques Analysis

Analysis disjunctive >= conjunctive >= phrase (DUH!) Flaws No tayloring of search to intent of query (by adding/excluding terms) or doing more complicated boolean expressions. No tayloring of search to specific capabilities of search engine (lowest common denominator)

Future Directions Use META tags to note content Add user feedback mechanisms Have small, specific databases, rather than monolithic databases Create common interfaces (federation of databases) Possibly allow better management of index content?