1. Evaluating Hierarchical Clustering of Search Results Departamento de Lenguajes y Sistemas Informáticos UNED, Spain Juan Cigarrán Anselmo Peñas Julio Gonzalo Felisa Verdejo nlp.uned.es SPIRE 2005, Buenos Aires

2. Overview Scenario Assumptions Features of a Good Hierarchical Clustering Evaluation Measures –Minimal Browsing Area (MBA) –Distillation Factor (DF) –Hierarchy Quality (HQ) Conclusion

3. Scenario Complex information needs –Compile information from different sources –Inspect the whole list of documents More than 100 documents Help to –Find the relevant topics –Discriminate from unrrelevant documents Approach –Hierarchical Clustering – Formal Concept Analysis

6. Problem How to define and measure the quality of a hierarchical clustering? How to compare different clustering approaches?

7. Previous assumptions Each cluster contains only those documents fully described by its descriptors d1d2d3d4 PhysicsXXXX Nuclear physics XX AstrophysicsX d2, d3 d1 Physics Astrophysics d4 Nuclear physics d2, d3 d1, d2, d3, d4 Physics Astrophysics d4 Nuclear physics

8. Previous assumptions ‘Open world’ perspective d1d2d3 PhysicsXX JokesXX Jokes about physics X d1 Physics Jokes about physics d3 Jokes d2 Jokes about physics d3 d1 Physics Jokes d2 Jokes about physics d3

9. Good Hierarchical Clustering The content of the clusters. –Clusters should not mix relevant with non relevant information + + + + + - - + - - + - + + - - +

10. Good Hierarchical Clustering The hierarchical arrangement of the clusters –Relevant information should be in the same path + + - - - + + - - - + + + + + - - - - -

11. Good Hierarchical Clustering The number of clusters –Number of clusters substantially lower than the number of documents How clusters are described –Cognitive load of reading a cluster description –Ability to predict the relevance of the information that it contains (not addressed here)

12. Evaluation Measures Criterion –Minimize the browsing effort for finding ALL relevant information Baseline –The original document list returned by a search engine

13. Evaluation Measures Consider –Content of clusters –Hierarchical arrangement of clusters –Size of the hierarchy –Cognitive load of reading a document (in the baseline): K d –Cognitive load of reading a node descriptor (in the hierarchy): K n Requirement –Relevance assessments are available

14. Minimal Browsing Area (MBA) The minimal set of nodes the user has to traverse to find ALL the relevant documents minimising the number of irrelevant ones + - -+ + + -

15. Distillation Factor (DF)  Ability to isolate relevant information compared with the original document list (Gain Factor, DF>1)  Considers only the cognitive load of reading documents  Equivalent to:

16. Distillation Factor (DF) Example DF(L) = 7/5 = 1.4 Doc 1+ Doc 2- Doc 3+ Doc 4+ Doc 5- Doc 6- Doc 7+ Document List Precision = 4/7Precision MBA = 4/5 + - -+ + + -

17. Distillation Factor (DF) Counterexample: + + -- + - - + Precision = 4/8 Precision MBA = 4/4 DF = 8/4 = 2 Bad clustering with good DF Extend the DF measure considering the cognitive cost of taking browsing decisions  HQ

18. Hierarchy Quality (HQ) Assumption: –When a node (in the MBA) is explored, all its lower neighbours have to be considered: some will be in turn explored, some will be discarded –N view : subset of lower neighbours of each node belonging to the MBA -+ - + + +- - - - MBA |N view |=8

19. Hierarchy Quality (HQ)  K n and K d are directly related with the retrieval scenario in which the experiments take place  The researcher must tune K=K n /K d before conducting the experiment  HQ > 1 indicates an improvement of the clustering versus the original list

20. Hierarchy Quality (HQ) -+ - + + +- - - - + + -- + - - + Example

21. Conclusions and Future Work Framework for comparing different clustering approaches taking into account: –Content of clusters –Hierarchical arrangement of clusters –Cognitive load to read document and node descriptions Adaptable to the retrieval scenario in which experiments take place Future work –Conduct user studies to compare their results with the automatic evaluation Results will reflect the quality of the descriptors Will be used to fine-tune the k d and k n parameters

22. Thank you!