IS530 Lesson 12 Boolean vs. Statistical Retrieval Systems

Boolean or Statistical? Most web search engines default to statistical, use Boolean for advanced Most proprietary online systems default to Boolean, use statistical for alternative Statistical search engine vs. relevance ranking of Boolean results

Web Search Engines Databases generated by robotic programs (non-human) spiders, wanderers, web walkers, agents Full-text indexing of website contents Supports advanced, complex search strategies

3 Parts of a Web Search Engine 1. Spider or web-crawler reads webpage, follows links 2. Index catalogs webpages read by spider 3. Search engine software matches queries lists most relevant site first

3 Parts of an Online System 1) Database building software (dataware) (follows rules with known fields) 2)Index/dictionary file (list of all words and sometimes phrases in the indexed fields) 3) Search engine software (matches queries; Boolean or statistical; LIFO or relevant

Boolean Operators AND limits search decreases hits increases precision OR expands search increases precision decreases hits NOT limits search seldom used too strong Proximity Operators Adj, (N)ear, (W)ith limit a search increase precision

Command Interface Boolean Searching (Westlaw) Find information about the assumption of risk involving people who fall after slipping in wintery conditions. assum! /5 risk / p (ic* or snow****) /p (slip! or fell or fall***)

Natural Language and Relevance Ranking (WIN) I need information on assumption of risk involving a person who has fallen on ice or snow.

Non-Boolean Retrieval Systems Statistical (associative, probabilistic, or relevance systems) Linguistic (semantic)

Statistical Retrieval Systems Incorporate relevance ranking May incorporate relevance feedback May have natural language interface Almost all web search engines use

Algorithm Latin algorismus, after al-KhwArizmi Arabian mathematician (AD 825) Step-by-step procedure for solving mathematical problems Merriam-Webster Statistical search engines use weighting algorithms to compute relevance

Statistical Search Engines Weighting algorithms are proprietary Search engines differ in how they assign weights and compute relevance ranking Search results differ studies found only about 40% overlap

Statistical Web Retrieval Factors Popularity, # other sites that link to a site authoritative sites given heavier weight Google Meta-tags may boost ranking Inktomi/Overture Direct hit may boost ranking HotBot

Linguistic Retrieval System Natural Language & Relevance Ranking WIN - (Westlaw Is Natural) has some elements I need information on assumption of risk involving a person who has fallen on ice or snow.

WIN Steps 1. Enter query in plain English 2. System removes stop phrases 3. Matches legal phrases from thesaurus, adjusts weighting 4. Removes stop words

WIN Steps (cont.) 5. Stemming 6. Searches database indexes in OR relationship 7. Statistical comparison applied 8. Results placed in ranked order

Factors in Determining Relevance Proximity of query words to each other Position of query words keywords in title rank higher keyword in headline or near top Relative length of document (“normalization”) Stemming

Factors in Determining Relevance (cont.) Ignore very frequent terms Inverse term frequency Relevance feedback Stop words Query expansion/thesaurus

Features Users Can Control Designating “bound phrases” Flagging terms that must be present* Specifying truncat? Indicating (synonym groups) Synonym dictionaries

Web Sites that list search engines and features: