Distributed PageRank Computation Based on Iterative Aggregation- Disaggregation Methods Yangbo Zhu, Shaozhi Ye and Xing Li Tsinghua University, Beijing, China ACM CIKM 2005, Bremen

Nov.2, Outline Quick Review of PageRank Distributed PageRank Computation Motivation Basic Idea Algorithm Experiments Conclusion and Future Work

Nov.2, PageRank - Background Ranking Web pages Content-based methods Link-based methods PageRank[Page & Brin, 1998] HITS[Kleinberg, 1998] SALSA[Lempel & Moran, 2000]

Nov.2, PageRank - Intuition Page A points to B means that the author of A recommends B. A page is of high quality if it is referred to by many other pages referred to by pages of high quality

Nov.2, PageRank - Model Random Surfer - Markov Chain

Nov.2, PageRank - Algorithm Power method

Nov.2, Outline Quick Review of PageRank Distributed PageRank Computation Motivation Basic Idea Algorithm Experiments Conclusion and Future Work

Nov.2, Motivation Compass search engine confederation

Nov.2, Motivation (cont.)

Nov.2, Basic Idea Divide and conquer Make use of the natural block structure of web graphs

Nov.2, DPC Algorithm Step 1 - Initialization Local nodes compute local PageRank vectors.

Nov.2, DPC Algorithm (cont.) Step 2 - Aggregation Central node computes the NodeRank vector.

Nov.2, DPC Algorithm (cont.) Step 3 - Disaggregation Local nodes compute extended local PageRank vectors. X: External nodes

Nov.2, DPC Algorithm (cont.) Step 4 - Central node computes the L1 distance between current global PageRank vector and previous one.

Nov.2, Advantages DPC mainly consists of standard PageRank computation. Small matrices fit into main memory. Low communication overhead.

Nov.2, Outline Quick Review of PageRank Distributed PageRank Computation Motivation Basic Idea Algorithm Experiments Conclusion and Future Work

Nov.2, Experimental Setup Simulation on a single Linux box. Group web pages by sites. For comparison Classic power method LPR-Ref-2 algorithm in [Wang, VLDB 2004]

Nov.2, Data Sets ST01/03 - crawled in 2001/2003 by Stanford WebBase Project CN04 - crawled in 2004 from web sites in China.

Nov.2, Evaluation Metrics L1 distance Kendall's τ-distance if page i and j are in different order in the two ranking lists.

Nov.2, Accuracy of the First Iteration L1 Kendall

Nov.2, Convergence Rate Number of iteration for convergence ( )

Nov.2, Outline Quick Review of PageRank Distributed PageRank Computation Experiments Conclusion and Future Work

Nov.2, Conclusion A distributed PageRank computation algorithm based on iterative aggregation- disaggregation (IAD) methods with Block Jacobi smoothing. Experiments on real web graphs show that DPC outperforms LPR-Ref-2[Wang, VLDB'04], and converges 5~7 times faster than Power method.

Nov.2, Future Work Implement DPC in distributed system. Integrate with Compass search engine confederation. How to update PageRank vectors efficiently within DPC framework?

Nov.2, Thank you !

Nov.2, General PageRank Algorithm

Nov.2, IAD Method - Notations Aggregation matrix(n×N) Disaggregation matrix(N×n)

Nov.2, IAD Method

Nov.2, DPC Algorithm

Nov.2, DPC Algorithm (Cont.)

Nov.2, DPC Algorithm (Cont.)

Nov.2, DPC - Convergence Analysis The global convergence of IAD method is still an open problem. The difficulty partly comes from that the disaggregation step is non-linear. The paper proves the global convergence of Block Jacobi method in PageRank scenario when n > 2.

Nov.2, Experiments - Basic Facts Distribution over size of sites Distribution over number of pages hosted by sites of different size

Nov.2, Experiments - Communication Overhead PowerLPR-Ref-2 / DPC Pos() - Number of positive elements L/U - Block strictly lower/upper triangular part of P