The web as a graph: structure and interpretation. Sridhar Rajagopalan IBM Almaden Ravi Kumar, Prabhakar Raghavan, Andrew Tomkins (IBM, Almaden) Andrei Broder, Farzin Maghoul (AltaVista Corp.) Raymie Stata, Janet Wiener (Compaq SRC) Eli Upfal (Brown University)
A Picture of (~200M) pages.
Part A: Structure. The graph. The Questions …. … and some Answers. The picture.
The web graph Nodes (or vertices) = web pages. Edges = (non-nepotistic) links. The graph = all web pages and links. Many nodes, estimates range from 500M to over 1B. Is very sparse. Average links/page between Average (links/page | more than 6 links) > 30. Concentrate on graph structure, ignore content.
Questions about the web graph How big is the graph?How many links on a page (outdegree)? How many links to a page (indegree)? Can one browse from any web page to any other? How many clicks? Can we pick a random page on the web? How different is browsing from a “random walk”? Can we exploit the structure of the web graph for searching and mining? What does the web graph reveal about social processes which result in its creation and dynamics?
Power laws: How many pages point to a random page on the web? Indegrees. Slope = 2.1
How many links on a page? Slope = 2.7
Yule/Pareto/Zipf and power laws. Inverse polynomial tail. Word frequency in text. Yule (later Mandelbrot) Statistical study of the literary vocabulary. [Yule, 1944]. Citation analysis [Lotka, 1926]. Zipf Human behavior and the principle of least effort. [Zipf, 1947]. Pareto Cours d’economie politique. [Pareto,1897] Network graph. [Faloutsos-Faloutsos-Faloutsos, 1999] Oligonucleotide sequences [Martindale-Konopka, 1996] Many other instances.
More Germane Access statistics for web pages. (From server logs) [Glassman- 97] User behavior (by instrumenting browsers and proxies) [Lukose- Huberman-98, Crovella and others,97-99] Earliest analytical model, [Herb Simon, 1955].
Co-citation and Bibliographic coupling: Signature of a community. Bipartite cores: small “complete” bipartite subgraphs. Bibliographic coupling, Co-citation analysis. Hubs and Authorities. Uses: Web searching (HITS/Clever). Mining communities (Campfire project). Backlink browsing, “find similar.”
Small world. (Small World Prediction) [Barabasi and Albert 99, Albert-Jeong- Barabasi 99]. Based on a simple model, predict that most pages are within 19 links of each other. Justify the model by crawling nd.edu
Facts (about the crawl). Most of the time (75%) a random page u is not reachable from another random page v. Indegree and Outdegree distributions satisfy the power law. Consistent over time and scale.
Component sizes. Component sizes are distributed by the power law.
Reachability How many vertices are reachable from a random vertex?
A Picture of (~200M) pages.
Part B: Interpretation Random graph theory. Application 1: The Campfire Project. Application 2: Classical IR/Learning.
Random Graphs Erdos and Renyi’s model [Bollobas]. Graph with n vertices. Each of n(n-1) arcs appear with probability p. Graphical evolution [Palmer]: study properties of the resulting random graph as p is increased from 0 to 1. [Shelah and Spencer] 0-1 law: Most properties exhibit a threshold “phase change” like behavior. p
Facts about the Erdos-Renyi model A random graph with average degree 4 has a giant connected component containing almost all (90%) of the vertices. Indegrees and outdegrees are concentrated around the mean. And have exponentially declining tails. Most vertices in the graph are close to most others (small world).
A new random graph model.
Content creation hypothesis Some page creators create content without regard to what exists on the web. Many create pages which are inspired by pre-existing content. Effectively, some links are random, others are copied from pre- existing pages.
Probabilistic analysis: Evolving graphs. Creation and Deletion processes for nodes and edges. –e.g. at each time step, a new node is created with a fixed probability –at each time step, a new edge is created with probability links two random nodes with probability a node in proportion to its indegree with probability (copy a random link). –At each time step a node (resp. edge) is deleted with probability (resp ) Simple model: creation probabilities are 1 and deletion probabilities are 0.
Theory
Why study models? Good predictors of macroscopic behavior. –Degree distributions. Existence and number of cores. [WWW8] Algorithmic advantages (speed and accuracy). –Better and analyzable algorithmic methods. Inclusion-Exclusion pruning. [VLDB]. –Applications to Data Mining. Better understanding of the data/corpus. –What is “surprising” depends on what is typical. To find interesting stuff, you must know what is expected.
Be careful about... Predicting and analyzing microscopic properties. – Microscopic Properties which can be changed by the addition/deletion of a few nodes/edges/features. Examples: Diameter and girth, rare terms and features. Very susceptible to noise and systematic but small inconsistencies in the model. – Macroscopic Major dataset surgery required to significantly alter the property. Examples: Degree distributions. Connectivity. Law of large numbers or equivalent applies.
Application 1: The campfire project.
Co-citation: Signature of a community. Bipartite cores: small “complete” bipartite subgraphs.
Campfire project Automatically find and organize communities on the web. Approach: –Find all cores. –Grow cores into the full community. –Do IR/Categorization/Clustering etc. to organize the community space. [KRRT] WWW8, and [KRRT] VLDB’99.
The cores are interesting. hotels in costa rica clipart japanese elementary schools turkish student associations oil spills off the coast of japan australian fire brigades aviation/aircraft vendors guitar manufacturers Yahoo!, Excite, Infoseek webrings news groups mailing lists Explicit communities. Implicit communities (1) Implicit communities are defined by cores. (2) There are an order of magnitute more of these. There are efficient heuristics to compute all cores. (3) Can grow the core to the community using Clever.
Costa Rican Hotels. The Costa Rica Inte...ion on arts, busi... Informatica Interna...rvices in Costa Rica Cocos Island Research Center Aero Costa Rica Hotel Tilawa - Home Page COSTA RICA BY tamarindo.com Costa Rica New Page 5 The Costa Rica Internet Directory. Costa Rica, Zarpe Travel and Casa Maria Si Como No Resort Hotels & Villas Apartotel El Sesteo... de San José, Cos... Spanish Abroad, Inc. Home Page Costa Rica's Pura V...ry - Reservation... YELLOW\RESPALDO\HOTELES\Orquide1 Costa Rica - Summary Profile COST RICA, MANUEL A...EPOS: VILLA Hotels and Travel in Costa Rica Nosara Hotels & Res...els & Restaurants... Costa Rica Travel, Tourism & Resorts Association Civica de Nosara Costa Rica, Healthy...t Pura Vida Domestic & International Airline HOTELES / HOTELS - COSTA RICA tourgems Hotel Tilawa - Links Costa Rica Hotels T...On line Reservations Yellow pages Costa...Rica Export INFOHUB Costa Rica Travel Guide Hotel Parador, Manuel Antonio, Costa Rica Destinations
Elementary Schools in Japan The American School in Japan The Link Page ‰ªès—§ˆä“c¬ŠwZƒz[ƒƒy[ƒW Kids' Space ˆÀés—§ˆÀé¼”¬ŠwZ ‹{鋳ˆç‘åŠw‘®¬ŠwZ KEIMEI GAKUEN Home Page ( Japanese ) Shiranuma Home Page fuzoku-es.fukui-u.ac.jp welcome to Miasa E&J school _“Þ쌧E‰¡ls—§’†ì¼¬ŠwZ‚̃y fukui haruyama-es HomePage Torisu primary school goo Yakumo Elementary,Hokkaido,Japan FUZOKU Home Page Kamishibun Elementary School... schools LINK Page-13 “ú–{‚ÌŠwZ a‰„¬ŠwZƒz[ƒƒy[ƒW 100 Schools Home Pages (English) K-12 from Japan 10/...rnet and Education ) ‚l‚f‚j¬ŠwZ‚U”N‚P‘g¨Œê ÒŠ—’¬—§ÒŠ—“Œ¬ŠwZ Koulutus ja oppilaitokset TOYODA HOMEPAGE Education Cay's Homepage(Japanese) –y“쬊wZ‚̃z[ƒƒy[ƒW UNIVERSITY ‰J—³¬ŠwZ DRAGON97-TOP ‰ª¬ŠwZ‚T”N‚P‘gƒz[ƒƒy[ƒW ¶µ°é¼ÂÁ© ¥á¥Ë¥å¡¼ ¥á¥Ë¥å¡¼
Application 2: Classical Learning/IR.
Vector space and other classical models. Document is a vector in a real-valued space with dimensions identified with “features.” [Salton] Some notion of similarity, usually, cosine or dot-product. Built in assumption: Features are independent.
Uses of the Vector Space model. Search, Clustering, Classification. Term weighting. [Salton, Dumais, Sparck-Jones] SVD (for instance, LSI [Deerwester et.al.]). Gaussian assumption and classification. (for instance, [Koller-Sahami], [Chakrabarti et.al.]). Many ad-hoc methods and heuristics, some of which work remarkably well.[Modha et.al.] Clustering. [Drineas et.al.] Dimensionality reduction. Feature selection. [Johnson-Lindenstrauss, Koller- Sahami, Chakrabarti et.al. and others]
Two (new ?) ingredients. Hypertext -- the graph. Zipfian distributions on term occurances.
Hypertext Classification/Clustering. Class of a page is a function of text + class of neighbor set. –Classification problem -- Markov Random fields. [Chakrabarti-Dom-Indyk] –Clustering problem -- [Modha]
Research Issue Rework applications in these new (rather old) context. OR Explain why the standard algorithms continue to work despite the sometime questionable assumptions behind their derivation.