1 Internet and Data Management Junghoo “John” Cho UCLA Computer Science

2 Legacy database Plain text files Biblio sever Information Galore

3 Challenges: Too much information? Discovery Discovery Management Management Overload Overload Access Access …

4 Approaches Central caching and Indexing Central caching and Indexing –Google, Excite, AltaVista Dynamic integration Dynamic integration –MySimon, BizRate

5 Central Caching and Indexing Central Index

6 Challenges Page selection and download Page selection and download –What page to download? Page and index update Page and index update –How to update pages? Page ranking Page ranking –What page is “important” or “relevant”? Scalability Scalability

7 Dynamic Integration Mediator Wrapper Source 1 Wrapper Source 2 Wrapper Source n

8 Heterogeneous sources Heterogeneous sources –Different data models: relational, object-oriented –Different schemas and representations: “Keanu Reeves” or “Reeves, K.” etc. Limited query capabilities Limited query capabilities Mediator caching Mediator caching Challenges

9 Outline of This Talk How can we maintain pages fresh? How does the Web change? How does the Web change? What do we mean by “fresh” pages? What do we mean by “fresh” pages? How should we refresh pages? How should we refresh pages?

10 Web Evolution Experiment How often does a Web page change? How often does a Web page change? How long does a page stay on the Web? How long does a page stay on the Web? How long does it take for 50% of the Web to change? How long does it take for 50% of the Web to change? How do we model Web changes? How do we model Web changes?

11 Experimental Setup February 17 to June 24, 1999 February 17 to June 24, sites visited (with permission) 270 sites visited (with permission) –identified 400 sites with highest “PageRank” –contacted administrators 720,000 pages collected 720,000 pages collected –3,000 pages from each site daily –start at root, visit breadth first (get new & old pages) –ran only 9pm - 6am, 10 seconds between site requests

12 Average Change Interval fraction of pages  average change interval 

13 Change Interval – By Domain fraction of pages   average change interval

14 Modeling Web Evolution Poisson process with rate Poisson process with rate T is time to next event T is time to next event f T (t) =  e - t (t > 0) f T (t) =  e - t (t > 0)

15 Change Interval of Pages for pages that change every 10 days on average interval in days fraction of changes with given interval Poisson model

16 Change Metrics Freshness Freshness –Freshness of element e i at time t is F ( e i ; t ) = 1 if e i is up-to-date at time t 0 otherwise eiei eiei... webdatabase Freshness of the database S at time t is F( S ; t ) = F( e i ; t ) (Assume “equal importance” of pages)  N 1 N i=1

17 Change Metrics Age Age –Age of element e i at time t is A( e i ; t ) = 0 if e i is up-to-date at time t t - (modification e i time) otherwise eiei eiei... webdatabase Age of the database S at time t is A( S ; t ) = A( e i ; t ) (Assume “equal importance” of pages)  N 1 N i=1

18 Change Metrics F(e i ) A(e i ) time update refresh Time averages:

19 Trick Question Two page database Two page database changes daily e 1 changes daily changes once a week e 2 changes once a week Can visit one page per week Can visit one page per week How should we visit pages? How should we visit pages? –... [uniform] –e 1 e 2 e 1 e 2 e 1 e 2 e 1 e 2... [uniform] – … [proportional] –e 1 e 1 e 1 e 1 e 1 e 1 e 1 e 2 e 1 e 1 … [proportional] –... –e 1 e 1 e 1 e 1 e 1 e 1... –... –e 2 e 2 e 2 e 2 e 2 e 2... –? e1e1 e2e2 e1e1 e2e2 web database

20 Proportional Often Not Good! Visit fast changing Visit fast changing e 1  get 1/2 day of freshness  get 1/2 day of freshness Visit slow changing Visit slow changing e 2  get 1/2 week of freshness Visiting e 2 is a better deal! Visiting e 2 is a better deal!

21 Optimal Refresh Frequency Problem Given       and f, find f 1, f 2,.., f N that maximize find f 1, f 2,.., f N that maximize

22 Optimal Refresh Frequency Shape of curve is the same in all cases Holds for any change frequency distribution

23 Optimal Refresh for Age Shape of curve is the same in all cases Holds for any change frequency distribution

24 Comparing Policies Based on Statistics from experiment and revisit frequency of every month

25 Not Every Page is Equal! e1e1 e2e2 Accessed by users 20 times/day Accessed by users 10 times/day Some pages are “more important” Some pages are “more important” In general, F (S ) = 1 F (e 1 ) + 2 F (e 2 )

26 Weighted Freshness w = 1 w = 2 f

27 Change Frequency Estimation How to estimate change frequency? How to estimate change frequency? –Naïve Estimator: X/T –X: number of detected changes –T: monitoring period –2 changes in 10 days: 0.2 times/day Change detected 1 day Page visited Page changed Incomplete change history Incomplete change history

28 Improved Estimator Based on the Poisson model Based on the Poisson model –X: number of detected changes –N: number of accesses –f : access frequency 3 changes in 10 days: 0.36 times/day  Accounts for “missed” changes

29 Improvement Significant? Application to a Web crawler Application to a Web crawler –Visit pages once every week for 5 weeks –Estimate change frequency –Adjust revisit frequency based on the estimate »Uniform: do not adjust »Naïve: based on the naïve estimator »Ours: based on our improved estimator

30 Improvement from Our Estimator Detected changes Ratio to uniform Uniform2,147, % 100% Naïve4,145,582193% Ours4,892,116228% (9,200,000 visits in total)

31 WebArchive Project Can we store the history of the Web? Can we store the history of the Web? –Web is ephemeral –Study of the Web evolution Challenges Challenges –Update? –Compression? –New storage? –Indexing?

32 Conclusion Exciting area and many challenges ahead! Exciting area and many challenges ahead! Thank you for your attention Thank you for your attention For more information visit For more information visithttp://