PRESENTED BY ASHISH CHAWLA AND VINIT ASHER The PageRank Citation Ranking: Bringing Order to the Web Lawrence Page and Sergey Brin, Stanford University

Agenda

Introduction Challenges in Information Retrieval on Web  Large # of documents  Heterogeneous and Unstructured  WWW  Is hypertext  provides auxiliary information (other than the text of web pages) Objective  Take advantage of this link structure.

Background Academic Citations  link to other well known papers  peer reviewed  have quality control Web :  Homogeneous in their quality, usage, citation & length  Quality measure (subjective to the user)  Importance of a page is a quantity that isn’t intuitively possible to capture

What does a user want? Most applicable documents first What is the job of a retrieval system?  Present more relevant documents upfront Notion: Quality/Importance of Web Pages  Difficult to classify (depends on user) We deal with the overall importance of a page, rather than individual sections of the page.

Link Structure Forward Links Back Links Web has 150 million pages and 1.7 billion links (probably more now) Use the concept of citation analysis  Highly linked pages are more “important" than pages with few links

Propagation of Ranking Page Rank: a page has high rank if the sum of the ranks of its back-links is high Some notations uWeb Page FuFu Set of pages u points to (Forward links) BuBu Set of pages that point to u (Backlinks) N u = |F u |Number of links from u cNormalization factor Simple Ranking function

Simplified Page Rank Calculation

Problem in Ranking? Rank Sink:  Two web pages that point to each other but to no other page. Third page which points to one of them.  loop will accumulate rank but never distribute it (since there are no outedges).

Page Rank Definition Let E(u) be some vector over the Web pages that corresponds to a source of rank. Then, the PageRank of a set of Web pages is an assignment, R’, to the Web pages which satisfies such that c is maximized and ||R’|| 1 = 1 (||R’|| 1 denotes the L 1 norm of R’).

Computing Page Rank initialize vector over web pages Loop: new ranks sum of normalized backlink ranks compute normalizing factor add escape term control parameter whilestop when converged

Random Surfer Model Random Surfer Clicks at random basis “Surfer” periodically gets bored.

Solution to Random Surfer Model Escape term: E(u) can be thought of as the random surfer gets bored periodically and jumps to a different page – not staying in the loop forever. We term this E to be a vector over all the web pages that accounts for each page’s escape probability (user defined parameter).

Another Problem – Dangling Links What are dangling links?  Links that point to any page with no outgoing links.  Pages not downloaded yet. Why is this a problem?  We don’t know how to distribute weight to these. What do we do ?  Remove them from the system

Mathematical Basics What is eigen vector and eigen value?  Given a vector v in the n-dimensional vector space, we can linear transform it to another vector space using a transformation matrix A. The transformed vector is Av.  An eigen vector is a vector that is scaled by a linear transformation, but not moved. The scaling factor is the eigen value. Eigen values and eigen vectors are not unique. We can compute them by Ax = x where is the eigen value of A and x is the corresponding eigen vector.  An eigenvector is a vector that 'points' in the same direction (has invariant direction cosines) under some transform. The eigenvalue is a number that describes how the magnitude of the eigenvector is scaled by the transform.

Mathematical Basics A is designated to be a matrix, u and v correspond to the columns of this matrix. Given that A is a matrix, and R be a vector over all the Web pages, the dominant eigenvector is the one associated with the maximal eigenvalue.

Example AT=AT=

Example (contd..) R = c A R = M R  c: eigenvalue  R : eigen vector of A A = R = Normalized = A x = λ x | A - λI | x = 0

Implementation Web crawler keeps a database of URLs so that it can discover all URLs on the web To implement PageRank, the web crawler builds an index of the URLs as it crawls Problems???  Infinitely large sites  Incorrect HTML  Sites are down  Web is always changing

PageRank Implementation Convert each URL into unique integer ID Link structure sorted by the IDs Remove dangling links Make a initial assignment of ranks and iterate until convergence Add the dangling links back Iterate the process again to assign weights to all dangling links Link database A, is normally kept in RAM

Convergence Properties PageRank will scale very well for large collections as the scaling factor is roughly linear in log n.

Convergence Properties Here we interpret web as a expander like graph. A graph is said to be expander if every subsets of nodes S has a neighborhood that is larger than some factor α times |S| Mathematically we verify the same if the largest eigenvalue is sufficiently larger than the second- largest eigenvalue

Searching with PageRank Two search engines implemented using PageRank.  Title based search engine  Matches titles of web pages with the given query  Ranks the results using PageRank  Works well for general queries having a large result set  Full text search engine (Google)  Scans the entire document for a match with the given query  Performs rank merging.

Types of Results Information based result  Finds a site which contains great deal of information  Propagates textual matching score through the link structure Common Case result  Most commonly used site (often commercial) relevant to the search query  PageRank results in good representation of the common case

Personalized PageRank E vector  Corresponds to a distribution of web pages  Provides flexibility in adjustment of PageRanks Uniform E causes highly linked web pages to achieve a very high ranking Single page E results in important pages not related to the homepage to achieve a low PageRank E consisting of root level pages of all web servers is a good compromise between uniform E and single page E

Applications Estimating Web Traffic  Looking at differences between PageRank and actual usage statistics, it is possible to find things that people often look at, but do not want to link to their web pages Backlink Predictor  Citation counts tends to get stuck in the local web pages  Using random surfer model, PageRank quickly finds the site homepage, and gives preference to its children resulting in an efficient, broad search  Hence PageRank potentially acts as a better backlink predictor since it builds up the entire website information faster

Other Applications Spam detection and prevention Sort the backlinks based on their importance

Issues Users are not random walkers. Starting point distribution (actual usage data as starting vector). Bias towards main pages. Linkage spam. No query specific rank.

Conclusion PageRank is a global ranking of all webpages, regardless of their content based solely on their location in the Web’s graph structure PageRank can be used to separate a small set of commonly used documents Full database is consulted only when small database is not adequate to answer the queries Personalized PageRank can be used to create a view of Web from a particular user’s perspective

Google Architecture..