The Structure of Broad Topics on the Web Soumen Chakrabarti Mukul M. Joshi Kunal Punera (IIT Bombay) David M. Pennock (NEC Research Institute)

Graph structure of the Web  Over two billion nodes, two trillion links  Power-law degree distribution Pr(degree = k)  1/k 2.1  Looks like a “bow-tie” at large scale IN OUT Strongly connected core (SCC) “This is the Web”

The need for content-based models  Why does a radius-1 expansion help in topic distillation?  Why does topic- specific focused crawling work?  Why is a global PageRank useful for specific queries? Search engine Query Root set Classifier Crawler Check frontier topic Prune if irrelevant Uniform jump Walk to out-neighbor

The need for content-based models  How are different topics linked to each other?  Are topic directories representative of Web topic populations?  Are standard collections (e.g., TREC W10G) representative of Web topics? “This is the Web with topics”

How to characterize “topics”  Web directories—most natural choice  Started with  Keep pruning until all leaf topics have enough (>300) samples  Approx 120k sample URLs  Flatten to approx 482 topics  Train text classifier (Rainbow)  Characterize new document d as a vector of probabilities p d = (Pr(c|d)  c) Classifier Test doc

Critique and defense  Cannot capture fine-grained or emerging topics Emerging topics most often specialize existing broad topics Broad topics rarely change  Classifier may be inaccurate Adequate if much better than random guessing of topic label Can compensate errors using held-out validation data

Background topic distribution  What fraction of Web pages are about Health?  Sampling via random walk PageRank walk (Henzinger et al.) Undirected regular walk (Bar- Yossef et al.)  Make graph undirected  Add self-loops so that all nodes have the same degree  Sample with large stride  Collect topic histograms

Convergence  Start from pairs of diverse topics  Two random walks, sample from each walk  Measure distance between topic distributions L 1 distance |p 1 – p 2 | =  c |p 1 (c) – p 2 (c)| in [0,2] Below.05 —.2 within 300—400 physical pages

Biases in topic directories  Use Dmoz to train a classifier  Sample the Web  Classify samples  Diff Dmoz topic distribution from Web sample topic distribution  Report maximum deviation in fractions  NOTE: Not exactly Dmoz

Topic-specific degree distribution  Preferential attachment: connect u to v w.p. proportional to the degree of v, regardless of topic  More realistic: u has a topic, and links to v with related topics  Unclear if power-law should be upheld Intra-topic linkage Inter-topic linkage

Random forward walk without jumps  Sampling walk is designed to mix topics well  How about walking forward without jumping? Start from a page u 0 on a specific topic Forward random walk (u 0, u 1, …, u i, …) Compare (Pr(c|u i )  c) with (Pr(c|u 0 )  c) and with the background distribution

 Forward walks wander away from starting topic slowly  But do not converge to the background distribution  Global PageRank ok also for topic-specific queries Jump parameter d=.1—.2 Topic drift not too bad within path length of 5—10 Prestige conferred mostly by same-topic neighbors  Also explains why focused crawling works Observations and implications W.p. d jump to a random node W.p. (1-d) jump to an out-neighbor u.a.r. High- prestige node Jump

Citation matrix  Given a page is about topic i, how likely is it to link to topic j? Matrix C[i,j] = probability that page about topic i links to page about topic j Soft counting: C[i,j] += Pr(i|u)Pr(j|v)  Applications Classifying Web pages into topics Focused crawling for topic-specific pages Finding relations between topics in a directory uv

Citation, confusion, correction From topic  True topic  From topic  To topic  Guessed topic  To topic  Arts Business Computers Games Health Home Recreation Reference Science Shopping Society Sports Classifier’s confusion on held-out documents can be used to correct confusion matrix

Fine-grained views of citation Clear block-structure derived from coarse-grain topics Strong diagonals reflect tightly-knit topic communities Prominent off-diagonal entries raise design issues for taxonomy editors and maintainers

Concluding remarks  A model for content-based communities New characterization and measurement of topical locality on the Web How to set the PageRank jump parameter? Topical stability of topic distillation Better crawling and classification  A tool for Web directory maintenance Fair sampling and representation of topics Block-structure and off-diagonals Taxonomy inversion