Web search results clustering Web search results clustering is a version of document clustering, but… Billions of pages Constantly changing Data mainly unstructured and heterogeneous Additional information to consider (i.e. links, click-through data, etc.)
Some requirements Fast – Immediate response to query Flexible – Web content changes constantly – overlapping User-oriented – Main goal is to aid the user in finding information – Meaningful labels – Visualization: GUI
Architecture of Web clustering engine
Main issues Online or offline clustering? What to use as input – Entire documents – Snippets – Structure information (links) – Other data (i.e. click-through) – Use stop word lists, stemming, etc. How to define similarity? – Content (i.e. vector-space model) – Link analysis – Usage statistics How to group similar documents? How to label the groups?
Clustering algorithms Flat or hierarchical? Overlapping? Hard or soft? (One object to one cluster or multiple clusters) Incremental? Predefined cluster number? Requiring explicit similarity measure? Distance measure?
Clustering algorithms Distance-based – Hierarchical Agglomerative Hierarchical Clustering (AHC) – Flat K-means (can be fuzzy) Single-pass (incremental) Other – Suffix Tree Clustering (Grouper) – Query directed clustering – Self-organizing (Kohonen) maps (neural networks) – Latent Semantic Indexing (LSI) (reducing the dimensionality of the vector-space)
Clustering algorithms Data Centric algorithms – E.g. Scatter/Gather – Vector space model, k-means or HAC – Cluster labels is not good enough Description-Aware algorithms – STC Description-Centric Algorithms – E.g Vivisimo, Lingo, SRC – Good labels.
Research Prototype systems
Commercial systems
Scatter/Gather from Lin and Pantel (2002) and Hearst and Pederson (1996)
KeySRC
KartOO
Others More at chnology.shtml
Scatter/Gather (Cutting et. al. 1992) Designed for browsing Based on two novel clustering algorithms – Buckshot – fast for online clustering – Fractionation – accurate for offline initial clustering of the entire set
Grouper (Zamir and Etzioni 1997, 1999) Online Operates on query result snippets Clusters together documents with large common subphrases Suffix Tree Clustering (STC): linear, incremental, overlapping, can be extended to hierarchical STC induces labeling
Iwona Białynicka-Birula - Clustering Web Search Results STC algorithm Step 1: Cleaning – Stemming – Sentence boundary identification – Punctuation elimination Step 2: Suffix tree construction – Produces base clusters (internal nodes) – Base clusters are scored based on size and phrase score (which depends on length and word „quality”) Step 3: Merging base clusters – Highly overlapping clusters are merged
Carrot 2 (Stefanowski and Weiss 2003) Component framework Allows substituting components for – Input (i.e. snippets from other search engines) – Filter Stemming Distance measure Clustering – Output
Vivísimo Commercial Online Hierarchical Conceptual
Iwona Białynicka-Birula - Clustering Web Search Results Other Mapuccino (IBM) – (Maarek et. al. 2000) – – Relatively efficient AHC (O(n 2 )) – Similarity based on vector-space model (Su et. al. 2001) – Only usage statistics used as input – Recursive Density Based Clustering SHOC – (Zhang and Dong 2004) – Grouper-like – Key phrase discovery
Problems The performance is far from being perfect – Incompleteness of clusters, Hard to tell why some cluster are generated, some are missing – Different cluster granularity, Some clusters are very specific, some are very broad – Inconsistency: the contents and the label, lack of intra and inter-cluster consistency. – Label expressiveness – Lack of evaluation, data sets
Future research trends To extract more powerful features: hyperlinks, external info, temporal attributes To generate more expressive or effective descriptions of clusters To improve the accuracy of the hierarchy structure To consider user characteristics, web usage data Integration with ontology Better visualisation of the clusters To apply to Mobile search XML documents clustering