1. 8/28/97Information Organization and Retrieval IR Implementation Issues, Web Crawlers and Web Search Engines University of California, Berkeley School of Information Management and Systems SIMS 202: Information Organization and Retrieval

2. 8/28/97Information Organization and Retrieval Review Boolean Retrieval Ranked Retrieval Vector Space Model

3. Information need Index Pre-process Parse Collections Rank or Match Query text input

4. 8/28/97Information Organization and Retrieval Boolean Model 3t33t3 1t11t1 2t22t2 1D11D1 2D22D2 3D33D3 4D44D4 5D55D5 6D66D6 8D88D8 7D77D7 9D99D9 10 D 10 11 D 11 m1m1 m2m2 m3m3 m5m5 m4m4 m7m7 m8m8 m6m6 m 2 = t 1 t 2 t 3 m 1 = t 1 t 2 t 3 m 4 = t 1 t 2 t 3 m 3 = t 1 t 2 t 3 m 6 = t 1 t 2 t 3 m 5 = t 1 t 2 t 3 m 8 = t 1 t 2 t 3 m 7 = t 1 t 2 t 3

5. 8/28/97Information Organization and Retrieval Boolean Searching “Measurement of the width of cracks in prestressed concrete beams” Formal Query: cracks AND beams AND Width_measurement AND Prestressed_concrete Cracks Beams Width measurement Prestressed concrete Relaxed Query: (C AND B AND P) OR (C AND B AND W) OR (C AND W AND P) OR (B AND W AND P)

6. 8/28/97Information Organization and Retrieval Boolean Problems Disjunctive (OR) queries lead to information overload Conjunctive (AND) queries lead to reduced, and commonly zero result Conjunctive queries imply reduction in Recall

7. 8/28/97Information Organization and Retrieval Advantages and Disadvantage of the Boolean Model Complete expressiveness for any identifiable subset of collection Exact and simple to program The whole panoply of Boolean Algebra available Advantages Complex query syntax is often misunderstood (if understood at all) Problems of Null output and Information Overload Output is not ordered in any useful fashion Disadvantages

8. 8/28/97Information Organization and Retrieval Boolean Extensions Fuzzy Logic –Adds weights to each term/concept –t a AND t b is interpreted as MIN(w(t a ),w(t b )) –t a OR t b is interpreted as MAX (w(t a ),w(t b )) Proximity/Adjacency operators –Interpreted as additional constraints on Boolean AND TOPIC system –Uses various weighted forms of Boolean logic and proximity information in calculating RSVs

9. 8/28/97Information Organization and Retrieval Vector Space Model Documents are represented as vectors in term space –Terms are usually stems –Documents represented by binary vectors of terms Queries represented the same as documents Query and Document weights are based on length and direction of their vector A vector distance measure between the query and documents is used to rank retrieved documents

10. 8/28/97Information Organization and Retrieval Documents in Vector Space t1t1 t2t2 t3t3 D1D1 D2D2 D 10 D3D3 D9D9 D4D4 D7D7 D8D8 D5D5 D 11 D6D6

11. 8/28/97Information Organization and Retrieval Vector Space Documents and Queries D1D1 D2D2 D3D3 D4D4 D5D5 D6D6 D7D7 D8D8 D9D9 D 10 D 11 t2t2 t3t3 t1t1

12. 8/28/97Information Organization and Retrieval Similarity Measures Simple matching (coordination level match) Dice’s Coefficient Jaccard’s Coefficient Cosine Coefficient Overlap Coefficient

13. 8/28/97Information Organization and Retrieval Vector Space with Term Weights and Cosine Matching 1.0 0.8 0.6 0.4 0.2 0.80.60.40.201.0 D2D2 D1D1 Q Term B Term A D i =(d i1,w di1 ;d i2, w di2 ;…;d it, w dit ) Q =(q i1,w qi1 ;q i2, w qi2 ;…;q it, w qit ) Q = (0.4,0.8) D1=(0.8,0.3) D2=(0.2,0.7)

14. 8/28/97Information Organization and Retrieval Problems with Vector Space There is no real theoretical basis for the assumption of a term space –it is more for visualization that having any real basis –most similarity measures work about the same regardless of model Terms are not really orthogonal dimensions –Terms are not independent of all other terms

15. 8/28/97Information Organization and Retrieval Today Probabilistic Retrieval (Introduction) Processing Ranked Queries (the role of inverted files) Web Crawlers - Distributed indexing of the WWW Probabilistic Retrieval (Details)

16. 8/28/97Information Organization and Retrieval Probabilistic Retrieval Goes back to 1960’s (Maron and Kuhns) Robertson’s “Probabilistic Ranking Principle” –Retrieved documents should be ranked in decreasing probability that they are relevant to the user’s query. –How to estimate these probabilities? Several methods (Model 1, Model 2, Model 3) with different emphases on how estimates are done.

17. 8/28/97Information Organization and Retrieval Probabilistic Models: Some Notation D = All present and future documents Q = All present and future queries (D i,Q j ) = A document query pair x = class of similar documents, y = class of similar queries, Relevance is a relation:

18. 8/28/97Information Organization and Retrieval Probabilistic Models Model 1 -- Probabilistic Indexing, P(R|y,D i ) Model 2 -- Probabilistic Querying, P(R|Q j,x) Model 3 -- Merged Model, P(R| Q j, D i ) Model 0 -- P(R|y,x) Probabilities are estimated based on prior usage or relevance estimation

19. 8/28/97Information Organization and Retrieval Probabilistic Models Rigorous formal model attempts to predict the probability that a given document will be relevant to a given query Ranks retrieved documents according to this probability of relevance (Probability Ranking Principle) Relies on accurate estimates of probabilities for accurate results

20. 8/28/97Information Organization and Retrieval Vector and Probabilistic Models Support “natural language” queries Treat documents and queries the same Support relevance feedback searching Support ranked retrieval Differ primarily in theoretical basis and in how the ranking is calculated –Vector assumes relevance –Probabilistic relies on relevance judgments or estimates

21. 8/28/97Information Organization and Retrieval Web Search Engines Most include some version of Vector Space or extended Boolean Some offer both “ranked” and Boolean, but not together. Some engines (such as those based on the original WAIS) are little more than coordination-level matching for ranked retrieval.

22. 8/28/97Information Organization and Retrieval Web Search Engines Some engines use added natural language processing techniques to identify concepts –Lycos based on work by Michael Mauldin at CMU –Excite’s “concept-based” search may be a development of Latent Semantic Indexing Some search engines using Probabilistic methods (with proprietary extensions) –Inktomi/HotBot uses a form of SLR.

23. 8/28/97Information Organization and Retrieval Web Search Engines Exact algorithms are not available for commercial WWW search engines Many search engines appear to be hybrids offering both ranked and Boolean elements

24. 8/28/97Information Organization and Retrieval Web Search Conclusions Web Search engines are stretching the performance limits of ranked retrieval algorithms Most Web search engines today attempt to combine the best features of ranked and Boolean searching There is still a long way to go before All and Only the Relevant web pages are retrieved in response to your query

25. 8/28/97Information Organization and Retrieval Web Crawlers How do the web search engines get all of the items they index? How do you store millions of words from hundreds of sites so that you can find them quickly (and efficiently)?

26. 8/28/97Information Organization and Retrieval Depth-First Crawling Page 1 Page 3 Page 2 Page 1 Page 2 Page 1 Page 5 Page 6 Page 4 Page 1 Page 2 Page 1 Page 3 Site 6 Site 5 Site 3 Site 1 Site 2

27. 8/28/97Information Organization and Retrieval Breadth First Page 1 Page 3 Page 2 Page 1 Page 2 Page 1 Page 5 Page 6 Page 4 Page 1 Page 2 Page 1 Page 3 Site 6 Site 5 Site 3 Site 1 Site 2

28. 8/28/97Information Organization and Retrieval Inverted Files We have seen “Vector files” conceptually, an Inverted File is a vector file “inverted” so that rows become columns and columns become rows

29. 8/28/97Information Organization and Retrieval How Are Inverted Files Created Documents are parsed to extract words (or stems) and these are saved with the Document ID. Now is the time for all good men to come to the aid of their country Doc 1 It was a dark and stormy night in the country manor. The time was past midnight Doc 2

30. 8/28/97Information Organization and Retrieval How Inverted Files are Created After all document have been parsed the inverted file is sorted

31. 8/28/97Information Organization and Retrieval How Inverted Files are Created Multiple term entries for a single document are merged and frequency information added

32. 8/28/97Information Organization and Retrieval How Inverted Files are Created The file is split into a Dictionary and a Postings file

33. 8/28/97Information Organization and Retrieval Inverted files Permit fast search for individual terms Search results for each term is a list of document IDs (and optionally, frequency and/or positional information) These lists can be used to solve Boolean queries: –country: d1, d2 –manor: d2 –country and manor: d2

34. 8/28/97Information Organization and Retrieval Inverted Files Lots of alternative implementations –E.g.: Cheshire builds within-document frequency using a hash table during parsing –Document IDs and frequency info are stored in a B-tree index keyed by the term. See the chapter on inverted files in the reader for other implementations.

35. 8/28/97Information Organization and Retrieval Probabilistic Models (Again) Rigorous formal model attempts to predict the probability that a given document will be relevant to a given query Ranks retrieved documents according to this probability of relevance (Probability Ranking Principle) Relies on accurate estimates of probabilities for accurate results

36. 8/28/97Information Organization and Retrieval Probabilistic Models: Logistic Regression Estimates for relevance based on log-linear model with various statistical measures of document content as independent variables. Log odds of relevance is a linear function of attributes: Term contributions summed: Probability of Relevance is inverse of log odds:

37. 8/28/97Information Organization and Retrieval Probabilistic Models: Logistic Regression attributes Average Absolute Query Frequency Query Length Average Absolute Document Frequency Document Length Average Inverse Document Frequency Inverse Document Frequency Number of Terms in common between query and document -- logged

38. 8/28/97Information Organization and Retrieval Probabilistic Models: Logistic Regression Probability of relevance is based on Logistic regression from a sample set of documents to determine values of the coefficients. At retrieval the probability estimate is obtained by: For the 6 X attribute measures shown previously

39. 8/28/97Information Organization and Retrieval Probabilistic Models Strong theoretical basis In principle should supply the best predictions of relevance given available information Can be implemented similarly to Vector Relevance information is required -- or is “guestimated” Important indicators of relevance may not be term -- though terms only are usually used Optimally requires on- going collection of relevance information AdvantagesDisadvantages