Predictive Ranking -H andling missing data on the web Haixuan Yang Group Meeting November 04, 2004

2 Outline Introduction Introduction Related Work Related Work Predictive Ranking Model Predictive Ranking Model Block Predictive Ranking Model Block Predictive Ranking Model Experiment Setup Experiment Setup

3 Introduction PageRank (1998) PageRank (1998) –It uses the link information to rank web page; –The importance of a page depends on the number of pages that point to it; –The importance of a page also depends on the importance of pages that point to it. –If X is the rank vector, Problems Problems –Manipulation –The “richer-get-richer” phenomenon –Computation Efficiency –Dangling nodes problem

4 Introduction Nodes that either have no out-link or for which no out-link is known are called dangling nodes. Nodes that either have no out-link or for which no out-link is known are called dangling nodes. Dangling nodes problem Dangling nodes problem –It is hard to sample the entire web. Page et al (1998) reported that they have 51 million URL not downloaded yet when they have 24 million pages downloaded.Page et al (1998) reported that they have 51 million URL not downloaded yet when they have 24 million pages downloaded. Handschuh et al (2003) estimated that dynamic pages are 100 times more than static pages.Handschuh et al (2003) estimated that dynamic pages are 100 times more than static pages. Eiron et al (2004) reported that the number of uncrawled pages (3.75 billion) still far exceeds the number of crawled pages (1.1 billion).Eiron et al (2004) reported that the number of uncrawled pages (3.75 billion) still far exceeds the number of crawled pages (1.1 billion). –Including dangling nodes in the overall ranking may not only change the rank value of non-dangling nodes but also change the order of them.

5 An example If we ignore the dangling node 3, then the ranks for nodes 1 and 2 are. If we consider the dangling node 3, then the ranks are by the revised pagerank algorithm (Kamvar 2003).

6 Introduction Classes of Dangling nodes Classes of Dangling nodes –Nodes that are found but not visited at current time are called dangling nodes of class 1. –Nodes that have been tried but not visited successfully are called dangling nodes of class 2. –Nodes, which have been visited successfully but from which no outlink is found, are called dangling nodes of class 3. Handle different kind of dangling nodes in different way. Our work focuses on dangling nodes of class 1, which cause missing information. Handle different kind of dangling nodes in different way. Our work focuses on dangling nodes of class 1, which cause missing information.

Illustration of dangling nodes At time 1 visited node:1. Dangling nodes of class 1: 2, 4, 5,7. At time 2, Dangling nodes of class 3 : 7 Visited nodes : 1,7,2; Dangling nodes of class 1: 3,4,5,6 77 Known information at time 2: red links Missing information at time 2: White links

8 Related work Page (1998): Simply removing them. After doing so, they can be added back in. The details are missing. Page (1998): Simply removing them. After doing so, they can be added back in. The details are missing. Amati (2003): Handle dangling nodes robustly based on a modified graph. Amati (2003): Handle dangling nodes robustly based on a modified graph. Kamvar (2003): Add uniform random jump from dangling nodes to all nodes. Kamvar (2003): Add uniform random jump from dangling nodes to all nodes. Eiron (2004): Speed up the model in Kamvar (2003), but sacrificing accuracy. Furthermore, suggest algorithm that penalize the nodes that link to dangling nodes of class 2. Eiron (2004): Speed up the model in Kamvar (2003), but sacrificing accuracy. Furthermore, suggest algorithm that penalize the nodes that link to dangling nodes of class 2.

9 Related work - Amati (2003)

10 Related work - Kamvar (2003)

11 Related work - Eiron (2004)

12 Predictive Ranking Model For dangling nodes of class 3, we use the same technique as Kamvar (2003). For dangling nodes of class 3, we use the same technique as Kamvar (2003). For dangling nodes of class 2, we ignore them at current model although it is possible to combine the push-back algorithm (Eiron 2004) with our model. (Penalizing nodes is a subjective matter.) For dangling nodes of class 2, we ignore them at current model although it is possible to combine the push-back algorithm (Eiron 2004) with our model. (Penalizing nodes is a subjective matter.) For dangling nodes of class 1, we try to predict the missing information about the link structrue. For dangling nodes of class 1, we try to predict the missing information about the link structrue.

13 Predictive Ranking Model Suppose that all the nodes V can be partitioned into three subsets:. Suppose that all the nodes V can be partitioned into three subsets:. – denotes the set of all nodes that have been crawled successfully and have at least one out-link; – denotes the set of all dangling nodes of class 3; – denotes the set of all dangling nodes of class 1; For each node in V, the real in-degree of v is not known. For each node in V, the real in-degree of v is not known. For each node v in, the real out-degree of v is known. For each node v in, the real out-degree of v is known. For each node v in, the real out-degree of v is known to be zero. For each node v in, the real out-degree of v is known to be zero. For each node v in, the real out-degree of v is unknown. For each node v in, the real out-degree of v is unknown.

14 Predictive Ranking Model We predict the real in-degree of v by the number of found links from C to v. We predict the real in-degree of v by the number of found links from C to v. –Assumption: the number of found links from C to v is proportional to the real number of links from V to v. For example, For example, if C and have 100 nodes, if C and have 100 nodes, V has 1000 nodes, V has 1000 nodes, and if the number of links from C to v is 5, and if the number of links from C to v is 5, then we estimate that the number of links from V to v is 50. then we estimate that the number of links from V to v is 50. The difference between these two numbers is distributed uniformly to the nodes in. The difference between these two numbers is distributed uniformly to the nodes in.

15 Predictive Ranking Model Models the missing information from unvisited nodes to nodes in V. Model the known link information as Page (1998): from C to V. Model the user’s behavior as Kamvar (2003) when facing dangling nodes of class 3.

16 Predictive Ranking Model Model users’ behavior (called as “teleportation”) as Page (1998) and Kamvar (2003) when the users get bored in following actual links and they may jump to some nodes randomly. is the rank vector.

17 Block Predictive Ranking Model Predict the in-degree of v more accurately. Predict the in-degree of v more accurately. Divide all nodes into p blocks (v[1], v[2], …, v[p]) according to their top level domains (for example, edu), or domains (for example, stanford.edu), or the countries (for example, cn). Divide all nodes into p blocks (v[1], v[2], …, v[p]) according to their top level domains (for example, edu), or domains (for example, stanford.edu), or the countries (for example, cn). Assumption: the number of found links from C[i] (C[i] is the meet of C and V[i]) to v is proportional to the real number of links from V[i] to v. Consequently, the matrix A is changed. Assumption: the number of found links from C[i] (C[i] is the meet of C and V[i]) to v is proportional to the real number of links from V[i] to v. Consequently, the matrix A is changed. Other parts are same as the Predictive Ranking Model. Other parts are same as the Predictive Ranking Model.

18 Block Predictive Ranking Model Models the missing information from unvisited nodes in 1st block to nodes in V.

19 Experiment Setup Get two datasets (by Patrick Lau): one is within the domain cuhk.edu.hk, the other is outside this domain. In the first dataset, we snapshot 11 matrices during the process of crawling; in the second dataset, we snapshot 9 matrices. Get two datasets (by Patrick Lau): one is within the domain cuhk.edu.hk, the other is outside this domain. In the first dataset, we snapshot 11 matrices during the process of crawling; in the second dataset, we snapshot 9 matrices. Apply both Predictive Ranking Model and the revised RageRank Model (Kamvar 2003) to these snapshots. Apply both Predictive Ranking Model and the revised RageRank Model (Kamvar 2003) to these snapshots. Compare the results of both models at time t with the future results of both models. Compare the results of both models at time t with the future results of both models. –The future results rank more nodes than the current results. So it is difficult to make a direct comparison.

20 Illustration for comparison future result current result Cut Normalize Comparison by 1-norm

21 Within domain cuhk.edu.hk Data Description Time t Vnum[t] Tnum[t] Time t Predictive Ranking Page Ranking Number of iterations

22 Within domain cuhk.edu.hk Comparison Based on future PageRank result at time 11.

23 Within domain cuhk.edu.hk Comparison Based on future PreRank result at time 11

24 Outside cuhk.edu.hk Data Description Time t Vnum[t] Tnum[t] Time t Predictive Ranking Page Ranking Number of iterations

25 Outside cuhk.edu.hk Comparison Based on future PageRank result at time 9

26 Outside cuhk.edu.hk Comparison Based on future PreRank result at time 9

27 Q & A