The Evolution of the Web and Implications for an Incremental Crawler Junghoo Cho Stanford University

What is a Crawler? web init get next url get page extract urls initial urls to visit urls visited urls web pages

Crawling Issues (1) Load at visited web sites Load at crawlers Scope of the crawl

Crawling Issues (2) Typical crawler Periodic, Batch, Shadowing Incremental crawling Maintain Pages “fresh” Avoid crawling from scratch How do we crawl?

Outline Web evolution experiments Freshness metrics Design issues and comparison

Web Evolution Experiment How often does a web page change? What is the lifespan of a page? How long does it take for 50% of the web to change?

Experimental Setup February 17 to June 24, sites visited (with permission) identified 400 sites with highest “page rank” contacted administrators 720,000 pages collected 3,000 pages from each site daily start at root, visit breadth first (get new & old pages) ran only 9pm - 6am, 10 seconds between site requests

How Often Does a Page Change? Example: 50 visits to page, 5 changes  average change interval = 50/5 = 10 days Is this correct? 1 day changes page visited

Average Change Interval fraction of pages

Average Change Interval — By Domain fraction of pages

How Long Does a Page Live? experiment duration page lifetime experiment duration page lifetime experiment duration page lifetime experiment duration page lifetime

Page Lifespans fraction of pages

Page Lifespans Method 1 used fraction of pages

Time for a 50% Change days fraction of unchanged pages

Change Metrics Freshness [SIGMOD 2000] Freshness of element e i at time t is F(e i ; t ) = 1 if e i is up-to-date at time t 0 otherwise eiei eiei... webdatabase  Freshness of the database S at time t is F(S ;t ) = F(e i ;t )  N 1 N i=1

Change Metrics Age [SIGMOD 2000] Age of element e i at time t is A(e i ; t ) = 0 if e i is up-to-date at time t t - (modification e i time) otherwise eiei eiei... webdatabase Age of the database S at time t is A(S ; t ) = A(e i ; t )  N 1 N i=1

Crawler Types In-place vs. shadow Steady vs. batch eiei eiei... webdatabase eiei... shadow database time crawler on crawler off

Comparison: Batch vs. Steady batch mode in-place crawler steady in-place crawler crawler running

Shadowing Steady Crawler crawler’s collection current collection without shadowing

Shadowing Batch Crawler crawler’s collection current collection without shadowing

Experimental Data: Freshness Pages change on average every 4 months Batch crawler works one week out of

Uniform vs. Variable In-place, steady crawler; Based on our experimental data [Pages change at different frequencies, as measured in experiment.] [SIGMOD 2000]

Summary Steady In-place Variable visit frequencies Improvement depends on on how the web changes improves freshness!

The End The paper proposes an architecture Thank you for your attention