1. WikiQuery.org -- An interactive collaboration interface for creating, storing and sharing effective CNF queries Le Zhao*, Xiaozhong Liu #, Jamie Callan* *: Language Tech Institute, SCS, Carnegie Mellon University # : School of Information, University of Indiana Bloomington @OSIR 2012, Portland, OR

2. Status Quo Current open source search engines – Good at: attracting software apps/service providers Lucene, Lemur, Terrier, … – Lacking: end users study users’ search behavior Two necessary conditions for success: – Attract users with a unique feature/functionality (not offered by current Web search engines) – Retain users not easily copied by the current search engines

3. This Work Unique opportunity – The term mismatch problem – Significant problem w/ huge potential [Zhao10,12] – Web search engines: automatic expansion – Lots of room for manual expansion Solution – Conjunctive normal form expansion – Commonly & effectively used by expert searchers [Lancaster68,Harter86,Hearst96,Baron07] 3

4. Average term mismatch rate: 30-40% [Zhao10] A common cause of search failure [Harman03, Zhao10] Frequent user frustration [Feild10] More than 50-300% gain in retrieval accuracy [Zhao12] Term Mismatch Problem 4 Relevant docs not returned Web, short queries, stemmed, inlinks included

5. Term Mismatch & Boolean Conjunctive Normal Form (CNF) Expansion Keyword query: approval of logos on television watched by children Manual CNF (TREC Legal track 2006): (approval OR guideline OR strategy) AND (logos OR promotion OR signage OR brand OR mascot OR marque OR mark) AND (television OR TV OR cable OR network) AND (watched OR view OR viewer) AND (children OR child OR teen OR juvenile OR kid OR adolescent) – Expressive & compact (1 CNF == 100s alternatives) – Highly effective (50-300% over base keyword [Zhao12]) – Widely used by experts (library, legal, medical …) – But, tedious to create 5

6. Goal of WikiQuery To facilitate the – creation (a proper tool is lacking for practitioners) – storing (for refinding & sharing) – and sharing (for collaboration on query parts) of high quality CNF queries 6

7. One Wiki Page == One Search Need

8. Displaying CNF Query

9. Links to Search Result Pages Relying on existing search engines for results

10. Creating/Editing CNF Queries

11. Search Result Access When Editing Interactions with search results are usually necessary to ensure quality of the created query.

12. Looks Similar, but Different from Library advanced search: simple Boolean – For example, Library of Congress Advance search: – Term1 AND/OR Term2 AND/OR Term3 LexisNexis: free form Boolean ERIC: more flexible, joining with AND or OR

13. Evaluating the WikiQuery Interface 13

14. Experiment Setup Hypothesis – Users (with limited knowledge about Boolean queries) can create effective Boolean queries using WikiQuery Preliminary user studies – Classroom users 6 students, limited prior knowledge of Boolean queries or IR a 10 minute session of editing Wiki pages – TREC topics => user Boolean CNF queries Total 12 topics, 12 final Boolean queries Interacted with Google and Yahoo (Spring, 2010) (40 minutes per topic) Evaluating on the Web & TREC collections 14

15. Results :-) – on average 30-50% gain over keyword queries :-( – But, not consistent (Web eval > TREC eval) – not statistically significant over strong kw baseline :-O – 75% contain more restrictive formulations, unstable – < 50% of queries are CNF expanded, bit more stable Need better tools to check quality of expansion terms 15

16. Conclusions Conjunctive Normal Form expansion has great potential Users need guidance and learning to create effective CNF queries – need to be warned against more restrictive queries Uses of WikiQuery: – Expert searchers – Classroom: becoming experts – Whenever you face a hard query 16

17. Source Code Based on MediaWiki version 1.17 Available at https://github.com/lezhao/wikiquery https://github.com/lezhao/wikiquery Questions: now OR catch me at lunch! 17