SLIDE 1IS 202 – FALL 2002 Lecture 20: Web Search Issues and Algorithms Prof. Ray Larson & Prof. Marc Davis UC Berkeley SIMS Tuesday and Thursday 10:30 am - 12:00 pm Fall SIMS 202: Information Organization and Retrieval

SLIDE 2IS 202 – FALL 2002 Lecture Overview Review –Evaluation of IR systems Precision vs. Recall Cutoff Points Test Collections/TREC Blair & Maron Study Web Crawling Web Search Engines and Algorithms Credit for some of the slides in this lecture goes to Marti Hearst

SLIDE 3IS 202 – FALL 2002 What can be measured that reflects users’ ability to use system? (Cleverdon 66) –Coverage of information –Form of presentation –Effort required/ease of use –Time and space efficiency –Recall Proportion of relevant material actually retrieved –Precision Proportion of retrieved material actually relevant What to Evaluate? Effectiveness

SLIDE 4IS 202 – FALL 2002 Precision vs. Recall Relevant Retrieved All Docs

SLIDE 5IS 202 – FALL 2002 Retrieved vs. Relevant Documents Very high precision, very low recall Relevant

SLIDE 6IS 202 – FALL 2002 Retrieved vs. Relevant Documents Very low precision, very low recall (0 in fact) Relevant

SLIDE 7IS 202 – FALL 2002 Retrieved vs. Relevant Documents High recall, but low precision Relevant

SLIDE 8IS 202 – FALL 2002 Retrieved vs. Relevant Documents High precision, high recall (at last!) Relevant

SLIDE 9IS 202 – FALL 2002 Precision/Recall Curves Difficult to determine which of these two hypothetical results is better: precision recall x x x x

SLIDE 10IS 202 – FALL 2002 Test Collections Cranfield 2 –1400 Documents, 221 Queries –200 Documents, 42 Queries INSPEC – 542 Documents, 97 Queries UKCIS – >10000 Documents, multiple sets, 193 Queries ADI – 82 Document, 35 Queries CACM – 3204 Documents, 50 Queries CISI – 1460 Documents, 35 Queries MEDLARS (Salton) 273 Documents, 18 Queries

SLIDE 11IS 202 – FALL 2002 TREC Text REtrieval Conference/Competition –Run by NIST (National Institute of Standards & Technology) –1999 was the 8th year - 9th TREC in early November Collection: >6 Gigabytes (5 CRDOMs), >1.5 Million Docs –Newswire & full text news (AP, WSJ, Ziff, FT) –Government documents (federal register, Congressional Record) –Radio Transcripts (FBIS) –Web “subsets” (“Large Web” separate with 18.5 Million pages of Web data – 100 Gbytes) –Patents

SLIDE 12IS 202 – FALL 2002 TREC (cont.) Queries + Relevance Judgments –Queries devised and judged by “Information Specialists” –Relevance judgments done only for those documents retrieved—not entire collection! Competition –Various research and commercial groups compete (TREC 6 had 51, TREC 7 had 56, TREC 8 had 66) –Results judged on precision and recall, going up to a recall level of 1000 documents Following slides from TREC overviews by Ellen Voorhees of NIST

SLIDE 13IS 202 – FALL 2002

SLIDE 14IS 202 – FALL 2002

SLIDE 15IS 202 – FALL 2002 Sample TREC Query (Topic) Number: 168 Topic: Financing AMTRAK Description: A document will address the role of the Federal Government in financing the operation of the National Railroad Transportation Corporation (AMTRAK) Narrative: A relevant document must provide information on the government’s responsibility to make AMTRAK an economically viable entity. It could also discuss the privatization of AMTRAK as an alternative to continuing government subsidies. Documents comparing government subsidies given to air and bus transportation with those provided to AMTRAK would also be relevant.

SLIDE 16IS 202 – FALL 2002

SLIDE 17IS 202 – FALL 2002

SLIDE 18IS 202 – FALL 2002

SLIDE 19IS 202 – FALL 2002

SLIDE 20IS 202 – FALL 2002

SLIDE 21IS 202 – FALL 2002 Blair and Maron 1985 A classic study of retrieval effectiveness –Earlier studies were on unrealistically small collections Studied an archive of documents for a legal suit –~350,000 pages of text –40 queries –Focus on high recall –Used IBM’s STAIRS full-text system Main Result: –The system retrieved less than 20% of the relevant documents for a particular information need –Lawyers thought they had 75% But many queries had very high precision

SLIDE 22IS 202 – FALL 2002 Blair and Maron (cont.) How they estimated recall –Generated partially random samples of unseen documents –Had users (unaware these were random) judge them for relevance Other results: –Two lawyers searches had similar performance –Lawyers recall was not much different from paralegal’s

SLIDE 23IS 202 – FALL 2002 Blair and Maron (cont.) Why recall was low –Users can’t foresee exact words and phrases that will indicate relevant documents “Accident” referred to by those responsible as: “event,” “incident,” “situation,” “problem,” … Differing technical terminology Slang, misspellings –Perhaps the value of higher recall decreases as the number of relevant documents grows, so more detailed queries were not attempted once the users were satisfied

SLIDE 24IS 202 – FALL 2002 Lecture Overview Review –Evaluation of IR systems Precision vs. Recall Cutoff Points Test Collections/TREC Blair & Maron Study Web Crawling Web Search Engines and Algorithms Credit for some of the slides in this lecture goes to Marti Hearst

SLIDE 25IS 202 – FALL 2002 Standard Web Search Engine Architecture crawl the web create an inverted index Check for duplicates, store the documents Inverted index Search engine servers user query Show results To user DocIds

SLIDE 26IS 202 – FALL 2002 Web Crawling How do the web search engines get all of the items they index? Main idea: –Start with known sites –Record information for these sites –Follow the links from each site –Record information found at new sites –Repeat

SLIDE 27IS 202 – FALL 2002 Web Crawlers How do the web search engines get all of the items they index? More precisely: –Put a set of known sites on a queue –Repeat the following until the queue is empty: Take the first page off of the queue If this page has not yet been processed: –Record the information found on this page »Positions of words, links going out, etc –Add each link on the current page to the queue –Record that this page has been processed In what order should the links be followed?

SLIDE 28IS 202 – FALL 2002 Page Visit Order Animated examples of breadth-first vs depth-first search on trees: – Structure to be traversed

SLIDE 29IS 202 – FALL 2002 Page Visit Order Animated examples of breadth-first vs depth-first search on trees: – Breadth-first search (must be in presentation mode to see this animation)

SLIDE 30IS 202 – FALL 2002 Page Visit Order Animated examples of breadth-first vs depth-first search on trees: – Depth-first search (must be in presentation mode to see this animation)

SLIDE 31IS 202 – FALL 2002 Page Visit Order Animated examples of breadth-first vs depth-first search on trees:

SLIDE 32IS 202 – FALL 2002 Sites Are Complex Graphs, Not Just Trees Page 1 Page 3 Page 2 Page 1 Page 2 Page 1 Page 5 Page 6 Page 4 Page 1 Page 2 Page 1 Page 3 Site 6 Site 5 Site 3 Site 1 Site 2

SLIDE 33IS 202 – FALL 2002 Web Crawling Issues Keep out signs –A file called robots.txt tells the crawler which directories are off limits Freshness –Figure out which pages change often –Recrawl these often Duplicates, virtual hosts, etc –Convert page contents with a hash function –Compare new pages to the hash table Lots of problems –Server unavailable –Incorrect html –Missing links –Infinite loops Web crawling is difficult to do robustly!

SLIDE 34IS 202 – FALL 2002 Searching the Web Web Directories versus Search Engines Some statistics about Web searching Challenges for Web Searching Search Engines –Crawling –Indexing –Querying

SLIDE 35IS 202 – FALL 2002 Directories vs. Search Engines Directories –Hand-selected sites –Search over the contents of the descriptions of the pages –Organized in advance into categories Search Engines –All pages in all sites –Search over the contents of the pages themselves –Organized after the query by relevance rankings or other scores

SLIDE 36IS 202 – FALL 2002 Search Engines vs. Internal Engines Not long ago HotBot, GoTo, Yahoo and Microsoft were all powered by Inktomi Today Google is the search engine behind many other search services (such as Yahoo)

SLIDE 37IS 202 – FALL 2002 Statistics from Inktomi Statistics from Inktomi, August 2000, for one client, one week –Total # queries: –Number of repeated queries: –Number of queries with repeated words: –Average words/ query: 2.39 –Query type: All words: ; Any words: ; Some words: –Boolean: ( AND / OR / NOT) –Phrase searches: –URL searches: –URL searches w/http: – searches: –Wildcards: ( '?'s ) frac '?' at end of query: interrogatives when '?' at end: composed of: –who: what: when: why: how: where where-MIS can,etc.: do(es)/did: 0.0

SLIDE 38IS 202 – FALL 2002 What Do People Search for on the Web? Topics –Genealogy/Public Figure:12% –Computer related:12% –Business:12% –Entertainment: 8% –Medical: 8% –Politics & Government 7% –News 7% –Hobbies 6% –General info/surfing 6% –Science 6% –Travel 5% –Arts/education/shopping/images 14% (from Spink et al. 98 study)

SLIDE 39 Visits (last year)

SLIDE 40IS 202 – FALL 2002 Visits (this year)

SLIDE 41 Directory Sizes (Last Year)

SLIDE 42IS 202 – FALL 2002 Directory Sizes

SLIDE 43IS 202 – FALL 2002

SLIDE 44IS 202 – FALL 2002

SLIDE 45 Searches Per Day (2000)

SLIDE 46IS 202 – FALL 2002 Searches Per Day (2001)

SLIDE 47IS 202 – FALL 2002 Searches per day (current) Couldn’t find exact numbers for Google, but they state in their “press” section that they handle 150 Million searches per day

SLIDE 48IS 202 – FALL 2002 Challenges for Web Searching: Data Distributed data Volatile data/”Freshness”: 40% of the web changes every month Exponential growth Unstructured and redundant data: 30% of web pages are near duplicates Unedited data Multiple formats Commercial biases Hidden data

SLIDE 49IS 202 – FALL 2002 Challenges for Web Searching: Users Users unfamiliar with search engine interfaces (e.g., Does the query “apples oranges” mean the same thing on all of the search engines?) Users unfamiliar with the logical view of the data (e.g., Is a search for “Oranges” the same things as a search for “oranges”?) Many different kinds of users

SLIDE 50IS 202 – FALL 2002 Web Search Queries Web search queries are SHORT –~2.4 words on average (Aug 2000) –Has increased, was 1.7 (~1997) User Expectations –Many say “the first item shown should be what I want to see”! –This works if the user has the most popular/common notion in mind

SLIDE 51IS 202 – FALL 2002 Search Engines Crawling Indexing Querying

SLIDE 52IS 202 – FALL 2002 Web Search Engine Layers From description of the FAST search engine, by Knut Risvik

SLIDE 53IS 202 – FALL 2002 Standard Web Search Engine Architecture crawl the web create an inverted index Check for duplicates, store the documents Inverted index Search engine servers user query Show results To user DocIds

SLIDE 54IS 202 – FALL 2002 More detailed architecture, from Brin & Page 98. Only covers the preprocessing in detail, not the query serving.

SLIDE 55IS 202 – FALL 2002 Indexes for Web Search Engines Inverted indexes are still used, even though the web is so huge Some systems partition the indexes across different machines –Each machine handles different parts of the data Other systems duplicate the data across many machines –Queries are distributed among the machines Most do a combination of these

SLIDE 56IS 202 – FALL 2002 Search Engine Querying In this example, the data for the pages is partitioned across machines. Additionally, each partition is allocated multiple machines to handle the queries. Each row can handle 120 queries per second Each column can handle 7M pages To handle more queries, add another row. From description of the FAST search engine, by Knut Risvik

SLIDE 57IS 202 – FALL 2002 Querying: Cascading Allocation of CPUs A variation on this that produces a cost- savings: –Put high-quality/common pages on many machines –Put lower quality/less common pages on fewer machines –Query goes to high quality machines first –If no hits found there, go to other machines

SLIDE 58IS 202 – FALL 2002 Google Google maintains the worlds largest Linux cluster (10,000 servers) These are partitioned between index servers and page servers –Index servers resolve the queries (massively parallel processing) –Page servers deliver the results of the queries Over 3 Billion web pages are indexed and served by Google

SLIDE 59IS 202 – FALL 2002 Search Engine Indexes Starting Points for Users include Manually compiled lists –Directories Page “popularity” –Frequently visited pages (in general) –Frequently visited pages as a result of a query Link “co-citation” –Which sites are linked to by other sites?

SLIDE 60IS 202 – FALL 2002 Starting Points: What is Really Being Used? Todays search engines combine these methods in various ways –Integration of Directories Today most web search engines integrate categories into the results listings Lycos, MSN, Google –Link analysis Google uses it; others are also using it Words on the links seems to be especially useful –Page popularity Many use DirectHit’s popularity rankings

SLIDE 61IS 202 – FALL 2002 Web Page Ranking Varies by search engine –Pretty messy in many cases –Details usually proprietary and fluctuating Combining subsets of: –Term frequencies –Term proximities –Term position (title, top of page, etc) –Term characteristics (boldface, capitalized, etc) –Link analysis information –Category information –Popularity information

SLIDE 62IS 202 – FALL 2002 Ranking: Hearst ‘96 Proximity search can help get high- precision results if >1 term –Combine Boolean and passage-level proximity –Proves significant improvements when retrieving top 5, 10, 20, 30 documents –Results reproduced by Mitra et al. 98 –Google uses something similar

SLIDE 63IS 202 – FALL 2002 Ranking: Link Analysis Assumptions: –If the pages pointing to this page are good, then this is also a good page –The words on the links pointing to this page are useful indicators of what this page is about –References: Page et al. 98, Kleinberg 98

SLIDE 64IS 202 – FALL 2002 Ranking: Link Analysis Why does this work? –The official Toyota site will be linked to by lots of other official (or high-quality) sites –The best Toyota fan-club site probably also has many links pointing to it –Less high-quality sites do not have as many high-quality sites linking to them

SLIDE 65IS 202 – FALL 2002 Ranking: PageRank Google uses the PageRank We assume page A has pages T1...Tn which point to it (i.e., are citations). The parameter d is a damping factor which can be set between 0 and 1. d is usually set to C(A) is defined as the number of links going out of page A. The PageRank of a page A is given as follows: PR(A) = (1-d) + d (PR(T1)/C(T1) PR(Tn)/C(Tn)) Note that the PageRanks form a probability distribution over web pages, so the sum of all web pages' PageRanks will be one

SLIDE 66IS 202 – FALL 2002 PageRank Similar to calculations used in scientific citation analysis (e.g., Garfield et al.) and social network analysis (e.g., Waserman et al.) Similar to other work on ranking (e.g., the hubs and authorities of Kleinberg et al.) Computation is an iterative algorithm and converges to the principle eigenvector of the link matrix

SLIDE 67IS 202 – FALL 2002 Next Time User Interfaces for Information Retrieval Readings: –Vannevar Bush, As We May Think ( computer/bushf.htm) –Don Norman, Why Interfaces Don’t Work –Modern Information Retrieval, Chapter 10 (Marti Hearst)