1. Crawling The Web For a Search Engine Or Why Crawling is Cool

2. Talk Outline What is a crawler? Some of the interesting problems RankMass Crawler As time permits: Refresh Policies Duplicate Detection

3. What is a Crawler? web init get next url get page extract urls initial urls to visit urls visited urls web pages

4. Applications Internet Search Engines Google, Yahoo, MSN, Ask Comparison Shopping Services Shopping Data mining Stanford Web Base, IBM Web Fountain

5. Is that it? Not quite

6. Crawling the Big Picture Duplicate Pages Mirror Sites Identifying Similar Pages Templates Deep Web When to stop? Incremental Crawler Refresh Policies Evolution of the Web Crawling the “good” pages first Focused Crawling Distributed Crawlers Crawler Friendly Webservers

7. Today’s Focus A crawler which guarantees coverage of the Web As time permits: Refresh Policies Duplicate Detection Techniques

8. RankMass Crawler A Crawler with High Personalized PageRank Coverage Guarantee

9. Motivation Impossible to download the entire web: Example: many pages from one calendar When can we stop? How to gain the most benefit from the pages we download

10. Main Issues Crawler Guarantee: guarantee on how much of the “important” part of the Web they “cover” when they stop crawling If we don’t see the pages, how do we know how important they are? Crawler Efficiency: Download “important” pages early during a crawl Obtain coverage with a min number of downloads

11. Outline Formalize coverage metric L-Neighbor: Crawling with RankMass guarantee RankMass: Crawling to achieve high RankMass Windowed RankMass: How greedy do you want to be? Experimental Results

12. Web Coverage Problem D – The potentially infinite set of documents of the web D C – The finite set of documents in our document collection Assign importance weights to each page

13. Web Coverage Problem What weights? Per query? Topic? Font? PageRank? Why PageRank? Useful as importance mesure Random surfer. Effective for ranking.

14. PageRank a Short Review p1 p2 p3 p4

15. Now it’s Personal Personal, TrustRank, General p3 p4

16. RankMass Defined Using personalized pagerank formally define RankMass of D C : Coverage Guarantee: We seek a crawler that given, when it stops the downloaded pages D C : Efficient crawling: We seek a crawler that, for a given N, downloads |D C |=N s.t. RM(D C ) is greater or equal to any other |D C |=N, D C D

17. How to Calculate RankMass Based on PageRank How do you compute RM(Dc) without downloading the entire web We can’t compute the exact but can lower bound Let’s a start a simple case

18. Single Trusted Page T (1) : t 1 =1 ; t i = 0 i≠1 Always jump to p 1 when bored We can place a lowerbound on being within L of P 1 N L (p 1 )=pages reachable from p 1 in L links

19. Single Trusted Page

20. Lower bound guarantee: Single Trusted Theorem 1: Assuming the trust vector T (1), the sum of the PageRank values of all L- neighbors of p1 is at least d L+1 close to 1.. That is:

21. Lower bound guarantee: General Case Theorem 2: The RankMass of the L-neighbors of the group of all trusted pages G, N L (G), is at least d L+1 close to 1. That is:

22. RankMass Lower Bound Lower bound given a single trusted page Extension: Given a set of trusted pages G That’s the basis of the crawling algorithm with a coverage guarantee

23. The L-Neighbor Crawler 1. L := 0 2. N[0] = {pi|ti > 0} // Start with the trusted pages 3. While ( < d L+1 ) 1. Download all uncrawled pages in N[L] 2. N[L + 1] = {all pages linked to by a page in N[L]} 3. L = L + 1

24. But what about efficency? L-Neighbor similar to BFS L-Neighbor simple and efficient May wish to prioritize further certain neighborhoods first Page level prioritization. t 0 =0.99 t 1 =0.01

25. Page Level Prioritizing We want a more fine-grained page-level priority The idea: Estimate PageRank on a page basis High priority for pages with a high estimate of PageRank We cannot calculate exact PageRank Calculate PageRank lower bound of undownloaded pages …But how

26. Probability of being at Page P Interrupted Page Random Surfer Click Link Trusted Page

27. Calculating PageRank Lower Bound PageRank(p) = Probability Random Surfer in p Breakdown path by “interrupts”, jumps to a trusted page Sum up all paths that start with an interrupt and end with p Interrupt PjP1P2P3P4P5Pi (1-d) (t j ) (d*1/3)(d*1/5) (d*1/3)

28. RankMass Basic Idea p 1 0.99 p 2 0.01 p 3 0.25 p 4 0.25 p 5 0.25 p 1 0.99 p 1 0.99 p 6 0.09 p 7 0.09 p 3 0.25 p 1 0.99

29. RankMass Crawler: High Level But that sounds complicated?! Luckily we don’t need all that Based on this idea: Dynamically update lower bound on PageRank Update total RankMass Download page with highest lower bound

30. RankMass Crawler (Shorter) Variables: CRM: RankMass lower bound of crawled pages rmi: Lower bound of PageRank of pi. RankMassCrawl() CRM = 0 rm i = (1 − d)t i for each t i > 0 While (CRM < 1 − ): Pick p i with the largest rm i. Download pi if not downloaded yet CRM = CRM + rm i Foreach p j linked to by p i : rm j = rm j + d/c i rm i rm i = 0

31. Greedy vs Simple L-Neighbor is simple RankMass is very greedy. Update expensive: random access to web graph Compromise? Batching downloads together updates together

32. Windowed RankMass Variables: CRM: RankMass lower bound of crawled pages rm i : Lower bound of PageRank of p i. Crawl() rmi = (1 − d)ti for each ti > 0 While (CRM < 1 − ): Download top window% pages according to rm i Foreach page p i ∈ D C CRM = CRM + rm i Foreach p j linked to by p i : rm j = rm j + d/c i rm i rm i = 0

33. Experimental Setup HTML files only Algorithms simulated over web graph Crawled between Dec’ 2003 and Jan’ 2004 141 millon URLs span over 6.9 million host names 233 top level domains.

34. Metrics Of Evaluation 1. How much RankMass is collected during the crawl 2. How much RankMass is “known” to have been collected during the crawl 3. How much computational and performance overhead the algorithm introduces.

35. L-Neighbor

36. RankMass

37. Windowed RankMass

38. Window Size

39. Algorithm Efficiency AlgorithmDownloads required for above 0.98% guaranteed RankMass Downloads required for above 0.98% actual RankMass L-Neighbor7 million65,000 RankMass131,07227,939 Windowed- RankMass 217,91830,826 Optimal27,101

40. Algorithm Running Time WindowHoursNumber of Iterations Number of Documents L-Neighbor1:271383,638,834 20%- Windowed 4:394480,622,045 10%- Windowed 10:278580,291,078 5%- Windowed 17:5216780,139,289 RankMass25:39Not comparable 10,350,000

41. Refresh Policies

42. Refresh Policy: Problem Definition You have N urls you want to keep fresh Limited resources: f documents / second Download order to maximize average freshness What do you do? Note: Can’t always know how the page really looks

43. The Optimal Solution Depends on freshness definition Freshness boolean: A page can only be fresh or not One small change deems it unfresh

44. Understand Freshness Better Two page database P d changes daily P w changes once a week We can refresh one page per week How should we visit pages? Uniform: P d, P d, P d, P d, P d, P d,… Proportional: P d,P d, P d, P d, P d,P d,P w Other?

45. Proportional Often Not Good! Visit fast changing e 1  get 1/2 day of freshness Visit slow changing e 2  get 1/2 week of freshness Visiting P w is a better deal!

46. Optimal Refresh Frequency Problem Given and f, find that maximize

47. Optimal Refresh Frequency Shape of curve is the same in all cases Holds for any change frequency distribution

48. 48 Do Not Crawl In The DUST: Different URLs Similar Text Ziv Bar-Yossef (Technion and Google) ‏ Idit Keidar (Technion) ‏ Uri Schonfeld (UCLA) ‏

49. 49 DUST – Different URLs Similar Text Examples: Default Directory Files: “/index.html”  “/” Domain names and virtual hosts “news.google.com”  “google.com/news” Aliases and symbolic links: “~shuri”  “/people/shuri” Parameters with little effect on content ?Print=1 URL transformations: “/story_ ”  “story?id= ” Even the WWW Gets Dusty

50. 50 Reduce the crawl and indexing Avoid fetching the same document more than once Canonization for better ranking References to a document may be split among its aliases Avoid returning duplicate results Many algorithm which use URLs as unique ids will benefit Why Care about DUST?

51. 51 Related Work Similarity detection via document sketches [Broder et al, Hoad-Zobel, Shivakumar et al, Di Iorio et al, Brin et al, Garcia-Molina et al] Requires fetching all document duplicates Cannot be used to find "DUST rules" Mirror detection [Bharat,Broder 99], [Bharat,Broder,Dean,Henzinger 00], [Cho,Shivakumar,Garcia-Molina 00], [Liang 01] Not suitable for finding site-specific "DUST rules" Mining association rules [Agrawal and Srikant] A technically different problem

52. 52 So what are we looking for?

53. 53 DustBuster, an algorithm that Discovers site-specific DUST rules from a URL list without examining page content Requires a small number of page fetches to validate the rules Site specific URL canonization algorithm Experimented on real data both from: Web access logs Crawl logs Our Contributions

54. 54 Valid DUST Rule: a mapping Ψ that maps each valid URL u to a valid URL Ψ(u) with similar content “/index.html”  “/” “news.google.com”  “google.com/news” “/story_ ”  “story?id= ” Invalid DUST Rules: Either do not preserve similarity or do not produce valid URLs 1 DUST Rules

55. 55  Substring Substitution DUST: “story_1259”  “story?id=1259” “news.google.com”  “google.com/news” “/index.html”  “” Parameter DUST: Removing a parameter or replacing its value to a default value “Color=pink”  “ Color=black ” Types of DUST Rules Focus of Talk

56. 56 Basic Detection Framework Input: List of URLs from a site: Crawl or web access log Detect likely DUST rules Eliminate redundant rules Validate DUST rules using samples DetectEliminateValidate No Fetch here

57. 57 Example: Instances & Support

58. END

59. RankMass Algorithm While ( ∑ i r i < 1 − ): Pick p i with the largest sumPathProb i. // Get the page with highest sumPathProbi. Download p i if not downloaded yet // Crawl the page. // Now expand all paths that end in pi PathsToExpand = Pop all paths ending with pi // Get all the paths leading to pi, from UnexploredPaths Foreach p j linked to from p i // and expand them by adding pi’s children to the paths. Foreach [path, prob] 2 PathsToExpand path′ = path · pj // Add the child pj to the path, prob′ = d/c ii prob // compute the probability of this expanded path, Push [path′, prob′] to UnexploredPaths // and add the expanded path to UnexploredPaths. sumPathProbj = sumPathProbj + d/ci sumPathProbi // Add the path probabilities of the newly added paths to pj. Add the probabilities of just explored paths to r i r i = r i + sumPathProb i // We just explored all paths to pi. Add their probabilities sumPathProbi = 0 // to r i.

60. RankMass Algorithm Variables: UnexploredPaths: List of unexplored paths and their path probabilities sumPathProb i : Sum of the probabilities of all unexplored paths leading to pi r i : Partial sum of the probability of being in pi RankMassCrawl() // Initialize: r i = 0 for each i // Set initial probability sum to be zero. UnexploredPaths = {} // Start with empty set of paths. Foreach (t i > 0): // Add initial paths of jumping to a trusted page and Push [path: {pi}, prob: (1 − d)t i ] to UnexploredPaths // the probability of the random jump. sumPathProb i = (1 − d)t i // For every trust page pi, we currently have only one path {pi}