1. In Situ Evaluation of Entity Ranking and Opinion Summarization using Kavita Ganesan & ChengXiang Zhai University of Illinois @ Urbana Champaign www.findilike.com

2. Preference – driven search engine – Currently works in hotels domain – Finds & ranks hotels based on user preferences: Structured: price, distance Unstructured: “friendly service”, “clean”, “good views” (Based on existing user reviews)  UNIQUE Beyond search: Support for analysis of hotels – Opinion summaries – Tag cloud visualization of reviews What is findilike?

3. …What is findilike? Developed as part of PhD. Work – new system (Opinion-Driven Decision Support System, UIUC, 2013) Tracked ~1000 unique users from Jan - Aug ‘13 – Working on speed & reaching out to more users

4. DEMO

5. 2 Components that can be evaluated through natural user interaction 1 Ranking entities based on unstructured user preferences Opinion-Based Entity Ranking (Ganesan & Zhai 2012) Ranking entities based on unstructured user preferences Opinion-Based Entity Ranking (Ganesan & Zhai 2012) Summarization of reviews Generating short phrases summarizing key opinions (Ganesan et. al 2010, 2012) Summarization of reviews Generating short phrases summarizing key opinions (Ganesan et. al 2010, 2012) 2

6. Evaluation of entity ranking Retrieval – Interleave results Balanced interleaving (T. Joachims, 2002) Balanced interleaving (T. Joachims, 2002) Base DirichletLM Base A click indicates preference…

7. Snapshot of pairwise comparison results for entity ranking AB C A > C B (A Better) C B > C A (B Better) C A = C B > 0 (Tie) C A = C B = 0Total DLMBase 30352572 PL2Base 10283748 …… …………… # Queries B is better Algorithms DirichletLM, Base, PL2 Algorithms DirichletLM, Base, PL2 # Queries A is Better

8. Snapshot of pairwise comparison results for entity ranking AB C A > C B (A Better) C B > C A (B Better) C A = C B > 0 (Tie) C A = C B = 0Total DLMBase 30352572 PL2Base 10283748 …… …………… Base model better & PL2 not too good Base model better, but DLM not too far behind

9. Evaluation of review summarization Randomly mix top N phrases from two algorithms Randomly mix top N phrases from two algorithms More clicks on phrases from Algo1 vs. Algo2  Algo1 better ALGO1 ALGO2 Monitor click- through on per entity basis

10. Submit code Performance report Online Performance AB C A > C B (A Better) C B > C A (B Better) C A = C B > 0 (Tie) DLMBase30352 PL2Base10283 …… ……… How to submit a new algorithm? Write Java based code Extend existing code Implementation

11. More information about evaluation… eval.findilike.com

12. Thanks! Questions? Links Evaluation: http://eval.findilike.comhttp://eval.findilike.com System: http://www.findilike.comhttp://www.findilike.com Related Papers: kavita-ganesan.comkavita-ganesan.com

13. References Ganesan, K. A., C. X. Zhai, and E. Viegas, Micropinion Generation: An Unsupervised Approach to Generating Ultra-Concise Summaries of Opinions, Proceedings of the 21st International Conference on World Wide Web 2012 (WWW '12), 2012.Micropinion Generation: An Unsupervised Approach to Generating Ultra-Concise Summaries of Opinions Ganesan, K. A., and C. X. Zhai, Opinion-Based Entity Ranking, Information Retrieval, vol. 15, issue 2, 2012Opinion-Based Entity Ranking Ganesan, K. A., C. X. Zhai, and J. Han, Opinosis: A Graph Based Approach to Abstractive Summarization of Highly Redundant Opinions, Proceedings of the 23rd International Conference on Computational Linguistics (COLING '10), 2010. Opinosis: A Graph Based Approach to Abstractive Summarization of Highly Redundant Opinions T. Joachims. Optimizing search engines using clickthrough data. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, KDD ’02, NY, 2002.

14. Evaluating Review Summarization Mini Test-bed Base code to extend Set of sample sentences Gold standard summary for those sentences ROUGE toolkit to evaluate the results Data set based on - Ganesan et. al 2010

15. Evaluating Entity Ranking Mini Test-bed Base code to extend Terrier Index of hotel reviews Gold standard ranking of hotels Code to generate nDCG scores. Raw unindexed data set for reference

16. Building a new ranking model Extend Weighting Model