A Scalable Semantic Indexing Framework for Peer-to-Peer Information Retrieval University of Illinois at Urbana-Champain Zhichen XuYan Chen Northwestern University ACM SIGIR HDIR 2005 Chengxiang Zhai Yahoo! Inc.

Motivation Rapid information growth requires scalable and robust retrieval architecture Problems with centralized retrieval architecture –Hard to maintain freshness of information –Single-point-of-failure Peer-to-Peer IR may be a possible solution –No need for centralized indexing –Easy to maintain freshness of information –Resistant to single-point-of-failure Challenge: P2P IR architecture?

Term Index vs. Document Index Term index –Fast query execution –Insufficient for supporting sophisticated algorithms such as feedback –Hard to update (e.g., adding a doc) in a distributed environment Document index –Easy to update –Support advanced retrieval algorithms –Slow query matching

What is the Right Indexing Architecture for P2P IR?

Previous Work: pSearch [Tang et al. 03] Based on document indexing Address the problem of “slow query execution” by –Dimension reduction (using LSI) –Exploiting distributed hash tables (DHT) Problems –Lack of semantic locality (semantically similar documents may be stored in quite different nodes) –Slow index generation –Hard to add a new concept

Proposed Solution: P2PIR, Scalable Semantic Indexing Framework for IR Semantic locality –Achieved through a novel two-phase distributed semantic indexing –Documents with similar semantics will have indices stored on nearby nodes Flexible tradeoff between search accuracy and efficiency Support of sophisticated retrieval methods –E.g., feedback and personalized search Adaptation to document dynamics –Incrementally incorporate new documents/concepts

Background on Sample DHT Content-Addressable Network A B CDE Two key operations Put (key, object) Object = get (key) Partition Cartesian space into zones Each zone is assigned to a computer Neighboring zones are routing neighbors Object lookup is done through routing

Routing and Location Properties on DHT Log(N) hops need to route a key where N is the number of nodes in the overlay Log(N) maintenance overhead for routing Guaranteed success Fault-tolerant and robust DoS attack resilient Becoming increasingly practical for serious use

P2PIR Architecture Two stage document indexing –Concept vector generation –Index locator generation and placement Open for plugging in “feature extraction”, “relevance ranking” and “query refinement” XML doc text doc feature extraction structure -aware feature vector term vector concept vector construction index locator construction index placement on DHT query Index locator generation search on DHT Relevance Ranking Applications P2PIR P2P DHT system Internet Results to user query refinement

Assumptions about the Retrieval Models Documents and queries are both represented as vectors –Naturally occurring in the vector-space model –Probabilistic models can be computed as vector matching as well Euclidean distances are reasonably accurate in capturing document topic similarity –Euclidean distances are only used to prune non- promising documents –Final relevance ranking can be based on more accurate retrieval functions

Concept Vector Construction Group document into k clusters based on the feature vectors The centroid of each cluster corresponds to a concept Given a document d, the similarity between its feature vector and a concept c (e.g., cosine value between them) defines the weight of d on concept c The concept vector of d is composed of its weights on all the concepts

Two-Stage Semantic Indexing Stage 1: Fast dimension reduction –Document clustering to identify n*d clusters (d= DHT dimension) –Represent each document with a vector on this n*d dimensional space Stage 2: Semantic index locator construction –Further partition the n*d clusters into n equal-size semantically coherent groups, each with size d –Each group forms an index locator (key for searching DHT)

Fast Dimension Reduction Regular k-means clustering –Randomly start with k centroids –Iteratively re-assign documents to each cluster and re-compute the centroids –Can stop at anytime to obtain rough clusters Modification –Start with k relatively different centroids Complexity at each iteration: O(kN), where N >>k is the number of documents Can be run on a sample of documents Vector(D)= (sim(D,C1), …, sim(D,C k ))

Index Locator Construction Motivation: the dimensionality of concept vectors (e.g., a few hundreds) may be much larger than that of DHT, so hard to place index directly with concept vector Basic idea: break the concept vector into multiple chunks with the same dimensionality as that of DHT, and each chunk contains related concepts With such division, each document only has a small number of chunks with non-negligible weights for indexing Such chunks are called index locators

Index Placement on DHT For each index locator of a document d If its norm (i.e., length of the vector) is over certain threshold, we put the index locator of d along with its feature vector on the peer node whose DHT address vector matches best with the index locator.

Illustration of Two-Stage Indexing D 1 D 2 … D N C 1 C 2 … C M Semantic Chunk 1 Semantic Chunk k Concepts C 1 C 2 … C M Doc D i = (x 1, x 2, …, x d, x d+1, …, x 2d, …. …. x M ) Locator 1Locator 2 …. (x 1, x 2, …, x d )  Original vector(D) (x 1, x 2, …, x d )  Original vector(D’) …… In DHT:

Querying Contact any node on the DHT Project the query vector to find related concepts, and form the index locators Use index locators to route to DHT nodes with the indices and feature vectors of related documents Use original query vector and document vectors to perform relevance ranking This local retrieval process can expand to neighboring DHT nodes until enough relevant results have been identified

Adaptation to Corpus Dynamics Basic idea: Incrementally add new documents/concepts without affecting existing indices, and periodically (very infrequently) rebuild index locators for all documents When a set of new documents emerge, we check 1.whether they contain new frequently-used terms or new heavy weighted terms 2.whether their concept vectors belong to any existing cluster in the existing semantic space

Adaptation to Corpus Dynamics (II) To add and index a new concept c –If c belongs to an existing concept chunk whose size is less than that of the underlying DHT, we can add c to that cluster by using the next available entry of the index locator. –Otherwise, we generate a new concept group and a new set of index locators to represent c Generate the index locators for the new documents, and deploy their indices on DHT Finally, multicast the addition of the new concept c, and the addition of new concept group to all DHT nodes, so that they can route queries about c

Example for Corpus Dynamics When new documents on “Bin Larden” appear, we detect it as a new concept relating to the concept group “terrorism”. If the dimensionality of DHT is 20, and the size of “terrorism” concept group is 17 –Just add “Bin Larden” to that group as dimension 18 of the index locator. –The corresponding index locators of existing documents have weight zero as default on dimension 18, and thus remain the same. Otherwise, the terrorism concept group already full, we generate a new concept group for “Bin Larden” (i.e., a new set of index locators).

Summary Propose a scalable semantic indexing framework for peer-to-peer information retrieval: P2PIR –Index placement with good semantic locality, leading to good retrieval accuracy and efficiency –Tunable framework and flexibility –Incremental adaptation to document/concept dynamics Prototype and evaluation of P2PIR in progress