1. Introduction to Information Retrieval Introduction to Information Retrieval Hinrich Schütze and Christina Lioma Lecture 19: Web Search 1

2. Introduction to Information Retrieval Overview ❶ Web characteristics ❷ Ads ❸ Index size and estimation ❹ Duplicate detection 2

3. Introduction to Information Retrieval Outline ❶ Web characteristics ❷ Ads ❸ Index size and estimation ❹ Duplicate detection 3

4. Introduction to Information Retrieval 4 Anchor text  Anchor text is often a better description of a page’s content than the page itself.  Anchor text can be weighted more highly than the text on the page. 4

5. Introduction to Information Retrieval 5 In-link and Out-link 5 A sample small web graph.In this example we have six pages labeled A-F. Page B has in-degree 3 and out- degree 1. This example graph is not strongly connected: there is no path from any of pages B-F to page A.

6. Introduction to Information Retrieval 6 Bow Tie 6 it is not possible to pass from a page in SCC to any page in IN, or from a page in OUT to a page in SCC. Most web pages fall into one of these three sets. The remaining pages form into tubes that are small sets of pages outside SCC that lead directly from IN to OUT, and tendrils that either lead nowhere from IN, or from nowhere to OUT.

7. Introduction to Information Retrieval 7 Spam 7 many web content creators have commercial motives and therefore stand to gain from manipulating search engine results. Search engines soon became sophisticated enough in their spam detection to screen out a large number of repetitions of particular keywords. Spammers responded with a richer set of spam techniques. The first of these techniques is cloaking. Here, the spammer's web server returns different pages depending on whether the http request comes from a web search engine's crawler.

8. Introduction to Information Retrieval 8 Search engine optimization (SEO) 8 Given that spamming is inherently an economically motivated activity, there has sprung around it an industry of Search Engine Optimizers, or SEOs to provide consultancy services for clients who seek to have their web pages rank highly on selected keywords

9. Introduction to Information Retrieval Outline ❶ Web characteristics ❷ Ads ❸ Index size and estimation ❹ Duplicate detection 9

10. Introduction to Information Retrieval 10 First generation of search ads: Goto (1996) 10  He paid $0.38 to Goto every time somebody clicked on the link.  Pages were simply ranked according to bid – revenue maximization for Goto.  No separation of ads/docs. Only one result list!

11. Introduction to Information Retrieval 11 Two ranked lists: web pages (left) and ads (right) 11

12. Introduction to Information Retrieval Algorithmic results. Paid Search Ads

13. Introduction to Information Retrieval 13 How are ads ranked?  First cut: according to bid price  Bad idea: open to abuse  Example: query [does my husband cheat?] → ad for divorce lawyer  We don’t want to show nonrelevant ads.  Instead: rank based on bid price and relevance  Result: A nonrelevant ad will be ranked low.  Even if this decreases search engine revenue short-term  Hope: Overall acceptance of the system and overall revenue is maximized if users get useful information. 13

14. Introduction to Information Retrieval Outline ❶ Web characteristics ❷ Ads ❸ Index size and estimation ❹ Duplicate detection 14

15. Introduction to Information Retrieval 15 The various components of a web search engine 15

16. Introduction to Information Retrieval 16 Estimates of the ratio of the index sizes of two search engines

17. Introduction to Information Retrieval Random searches Choose random searches extracted from a local log [Lawrence & Giles 97] or build “random searches” list. Use only queries with small result sets. Count normalized URLs in result sets. Sec. 19.5

18. Introduction to Information Retrieval adaptive access control neighborhood preservation topographic hamiltonian structures right linear grammar pulse width modulation neural unbalanced prior probabilities ranked assignment method internet explorer favourites importing karvel thornber zili liu Queries from Lawrence and Giles study softmax activation function bose multidimensional system theory gamma mlp dvi2pdf john oliensis rieke spikes exploring neural video watermarking counterpropagation network fat shattering dimension abelson amorphous computing Sec. 19.5

19. Introduction to Information Retrieval Random IP addresses Generate random IP addresses Find a web server at the given address – If there’s one Collect all pages from server – From this, choose a page at random – The biases here include the fact that many hosts might share one IP (due to a practice known as virtual hosting) or not accept http requests from the host. Sec. 19.5

20. Introduction to Information Retrieval Random walks View the Web as a directed graph Build a random walk on this graph – Includes various “jump” rules back to visited sites Does not get stuck in spider traps! Can follow all links! – Converges to a stationary distribution Must assume graph is finite and independent of the walk. Time to convergence not really known Sec. 19.5

21. Introduction to Information Retrieval Outline ❶ Web characteristics ❷ Ads ❸ Index size and estimation ❹ Duplicate detection 21

22. Introduction to Information Retrieval 22 Duplicate detection  The web is full of duplicated content.  More so than many other collections  Exact duplicates  Easy to eliminate  E.g., use hash/fingerprint  Near-duplicates  Abundant on the web  Difficult to eliminate  For the user, it’s annoying to get a search result with near-identical documents. 22

23. Introduction to Information Retrieval 23 Near-duplicates: Example 23

24. Introduction to Information Retrieval 24 Detecting near-duplicates  Compute similarity with an edit-distance measure  We want “syntactic” (as opposed to semantic) similarity.  True semantic similarity (similarity in content) is too difficult to compute.  We do not consider documents near-duplicates if they have the same content, but express it with different words.  Use similarity threshold θ to make the call “is/isn’t a near- duplicate”.  E.g., two documents are near-duplicates if similarity > θ = 80%. 24

25. Introduction to Information Retrieval 25 Represent each document as set of shingles  A shingle is simply a word n-gram.  Shingles are used as features to measure syntactic similarity of documents.  For example, for n = 3, “a rose is a rose is a rose” would be represented as this set of shingles:  { a-rose-is, rose-is-a, is-a-rose }  We can map shingles to a hash value over a large space,say 64bits.  We define the similarity of two documents as the Jaccard coefficient of their shingle sets. 25

26. Introduction to Information Retrieval 26 Recall: Jaccard coefficient  A commonly used measure of overlap of two sets  Let A and B be two sets  Jaccard coefficient:  JACCARD (A,A) = 1  JACCARD (A,B) = 0 if A ∩ B = 0  A and B don’t have to be the same size.  Always assigns a number between 0 and 1. 26

27. Introduction to Information Retrieval 27 Jaccard coefficient: Example  Three documents: d 1 : “Jack London traveled to Oakland” d 2 : “Jack London traveled to the city of Oakland” d 3 : “Jack traveled from Oakland to London”  Based on shingles of size 2 (2-grams or bigrams), what are the Jaccard coefficients J(d 1, d 2 ) and J(d 1, d 3 )?  J(d 1, d 2 ) = 3/8 = 0.375  J(d 1, d 3 ) = 0  Note: very sensitive to dissimilarity 27

28. Introduction to Information Retrieval 28 Represent each document as a sketch  The number of shingles per document is large.  To increase efficiency, we will use a sketch, a cleverly chosen subset of the shingles of a document.  The size of a sketch is, say, n = 200... ... and is defined by a set of permutations π 1... π 200.  Each π i is a random permutation on 1..2 m  The sketch of d is defined as: (a vector of 200 numbers). 28

29. Introduction to Information Retrieval 29 The Permutation and minimum: Example document 1: {s k } document 2: {s k } We use min s ∈ d1 π(s) = min s ∈ d 2 π(s) as a test for: are d 1 and d 2 near-duplicates? In this case: permutation π says: d 1 ≈ d 2 29

30. Introduction to Information Retrieval 30 Computing Jaccard for sketches  Let U be the union of the set of shingles of d 1 and d 2 and I the intersection.  There are |U|! permutations on U.  For s′ ∈ I, for how many permutations π do we have argmin s ∈ d1 π(s) = s′ = argmin s ∈ d2 π(s)?  Answer: (|U| − 1)!  There is a set of (|U| − 1)! different permutations for each s in I. ⇒ |I |(|U| − 1)! permutations make argmin s ∈ d1 π(s) = argmin s ∈ d2 π(s) true  Thus, the proportion of permutations that make min s ∈ d1 π(s) = min s ∈ d2 π(s) true is: 30

31. Introduction to Information Retrieval 31 Estimating Jaccard  Thus, the proportion of successful permutations is the Jaccard coefficient.  Permutation π is successful iff min s ∈ d1 π(s) = min s ∈ d2 π(s)  Picking a permutation at random and outputting 1 (successful) or 0 (unsuccessful) is a Bernoulli trial.  Estimator of probability of success: proportion of successes in n Bernoulli trials. (n = 200)  Our sketch is based on a random selection of permutations.  Thus, to compute Jaccard, count the number k of successful permutations for and divide by n = 200. k/n = k/200 estimates J(d 1, d 2 ). 31

32. Introduction to Information Retrieval 32 Implementation  We use hash functions as an efficient type of permutation: h i : {1..2 m } → {1..2 m }  Scan all shingles s k in union of two sets in arbitrary order  For each hash function h i and documents d 1, d 2,...: keep slot for minimum value found so far  If h i (s k ) is lower than minimum found so far: update slot 32

33. Introduction to Information Retrieval 33 Exercise h(x) = 5x + 5 mod 4 g(x) = (3x + 1) mod 4 33

34. Introduction to Information Retrieval 34 Solution final sketches 34 h(x) = 5x + 5 mod 4 g(x) = (3x + 1) mod 4

35. Introduction to Information Retrieval 35 Solution 35

36. Introduction to Information Retrieval 36 Shingling: Summary  Input: N documents  Choose n-gram size for shingling, e.g., n = 5  Pick 200 random permutations, represented as hash functions  Compute N sketches: 200 × N matrix,one row per permutation, one column per document  Compute pairwise similarities  Transitive closure of documents with similarity > θ  Index only one document from each equivalence class 36