Looking at both the Present and the Past to Efficiently Update Replicas of Web Content Luciano Barbosa * Ana Carolina Salgado! Francisco Tenorio! Jacques Robin! Juliana Freire * *Univesity of Utah ! Universidade Federal de Pernambuco

WIDM Looking at both the Present and the Past to Efficiently Update Replicas of Web Content Keeping Web Information Up-to-date Types of applications Proxy servers Search engine Quality of results: broken links

WIDM Looking at both the Present and the Past to Efficiently Update Replicas of Web Content Keeping Web Information Up-to-date Challenges in Web Data Atualization Sources autonomous and independent Lots of data billion of pages Dynamism 40% of Web pages change at least once a week (Cho and Molina, 2000) Applications run over limited resources Search engine coverage – 42% (Lawrence and Giles, 1999) Average time to search engeine updates a page – 186 days

WIDM Looking at both the Present and the Past to Efficiently Update Replicas of Web Content Updating Web Content: Our Solution Basic Idea Predict change rate of pages Update pages based on this prediction Two phases First visit Page attributes, e.g., file size and number of pages Over time Change history

WIDM Looking at both the Present and the Past to Efficiently Update Replicas of Web Content Updating Web Content: Our Solution Crawler New Page? Historic Classifier Static Classifier Change History Change Predictions PageYes No Change prediction Change prediction Page history

WIDM Looking at both the Present and the Past to Efficiently Update Replicas of Web Content Building the Solution: Overview 1. Gathering the training set 2. Creating the change rate groups 3. Learning static features 4. Learning from History

WIDM Looking at both the Present and the Past to Efficiently Update Replicas of Web Content Gathering the Training Set 100 most accessed sites of the Brazilian Web Breadth-first search down to depth 9 Total of URLs Each page visited once a day for 100 days

WIDM Looking at both the Present and the Past to Efficiently Update Replicas of Web Content Creating the Change Rate Groups Predict the average interval of time at which a given page is modified Regression task Discretizing the target attribute Classification task Performed an unsupervised discretization Result:

WIDM Looking at both the Present and the Past to Efficiently Update Replicas of Web Content Learning Static Features Relation between some Web page attributes in its dynamism Dynamic pages are larger and have more images [Douglis et al] The absence of the HTTP header LAST-MODIFIED indicates that a page is more volatile than pages that contain this information [Brewington and Cybenko]

WIDM Looking at both the Present and the Past to Efficiently Update Replicas of Web Content Learning Static Features Attributes used: Number of links Number of addresses Existence of the HTTP header LAST-MODIFIED File size in bytes (without html tags) Number of images Depth of a page in its domain A domain represents, for instance, for the site every page in *.yahoo.com) The directory level of the page URL in relation to the URL root from the Web server E.g., is level 1, level 2, and so on).

WIDM Looking at both the Present and the Past to Efficiently Update Replicas of Web Content Determining the Relevance of the Features Feature selection task Wrapper method with backward elimination Result Depth of a page in its domain is not relevant Remaining features used in the static classifier

WIDM Looking at both the Present and the Past to Efficiently Update Replicas of Web Content Building the Static Classifier Classification algorithms J48: decision tree NaiveBayes: naïve bayes IBk: k-nearest neighbor Measures of performance Error rate Classification time Results AlgorithmsError test rateClassification time J48 without pruning s J48 postpruning s NaivesBayes s IBk with k= s IBk with k= s

WIDM Looking at both the Present and the Past to Efficiently Update Replicas of Web Content Learning from History

WIDM Looking at both the Present and the Past to Efficiently Update Replicas of Web Content Learning from History

WIDM Looking at both the Present and the Past to Efficiently Update Replicas of Web Content Experimental Results

WIDM Looking at both the Present and the Past to Efficiently Update Replicas of Web Content Experimental Results

WIDM Looking at both the Present and the Past to Efficiently Update Replicas of Web Content Future Directions