WikiQuery.org -- An interactive collaboration interface for creating, storing and sharing effective CNF queries Le Zhao*, Xiaozhong Liu #, Jamie Callan* *: Language Tech Institute, SCS, Carnegie Mellon University # : School of Information, University of Indiana 2012, Portland, OR

Status Quo Current open source search engines – Good at: attracting software apps/service providers Lucene, Lemur, Terrier, … – Lacking: end users study users’ search behavior Two necessary conditions for success: – Attract users with a unique feature/functionality (not offered by current Web search engines) – Retain users not easily copied by the current search engines

This Work Unique opportunity – The term mismatch problem – Significant problem w/ huge potential [Zhao10,12] – Web search engines: automatic expansion – Lots of room for manual expansion Solution – Conjunctive normal form expansion – Commonly & effectively used by expert searchers [Lancaster68,Harter86,Hearst96,Baron07] 3

Average term mismatch rate: 30-40% [Zhao10] A common cause of search failure [Harman03, Zhao10] Frequent user frustration [Feild10] More than % gain in retrieval accuracy [Zhao12] Term Mismatch Problem 4 Relevant docs not returned Web, short queries, stemmed, inlinks included

Term Mismatch & Boolean Conjunctive Normal Form (CNF) Expansion Keyword query: approval of logos on television watched by children Manual CNF (TREC Legal track 2006): (approval OR guideline OR strategy) AND (logos OR promotion OR signage OR brand OR mascot OR marque OR mark) AND (television OR TV OR cable OR network) AND (watched OR view OR viewer) AND (children OR child OR teen OR juvenile OR kid OR adolescent) – Expressive & compact (1 CNF == 100s alternatives) – Highly effective (50-300% over base keyword [Zhao12]) – Widely used by experts (library, legal, medical …) – But, tedious to create 5

Goal of WikiQuery To facilitate the – creation (a proper tool is lacking for practitioners) – storing (for refinding & sharing) – and sharing (for collaboration on query parts) of high quality CNF queries 6

One Wiki Page == One Search Need

Displaying CNF Query

Links to Search Result Pages Relying on existing search engines for results

Creating/Editing CNF Queries

Search Result Access When Editing Interactions with search results are usually necessary to ensure quality of the created query.

Looks Similar, but Different from Library advanced search: simple Boolean – For example, Library of Congress Advance search: – Term1 AND/OR Term2 AND/OR Term3 LexisNexis: free form Boolean ERIC: more flexible, joining with AND or OR

Evaluating the WikiQuery Interface 13

Experiment Setup Hypothesis – Users (with limited knowledge about Boolean queries) can create effective Boolean queries using WikiQuery Preliminary user studies – Classroom users 6 students, limited prior knowledge of Boolean queries or IR a 10 minute session of editing Wiki pages – TREC topics => user Boolean CNF queries Total 12 topics, 12 final Boolean queries Interacted with Google and Yahoo (Spring, 2010) (40 minutes per topic) Evaluating on the Web & TREC collections 14

Results :-) – on average 30-50% gain over keyword queries :-( – But, not consistent (Web eval > TREC eval) – not statistically significant over strong kw baseline :-O – 75% contain more restrictive formulations, unstable – < 50% of queries are CNF expanded, bit more stable Need better tools to check quality of expansion terms 15

Conclusions Conjunctive Normal Form expansion has great potential Users need guidance and learning to create effective CNF queries – need to be warned against more restrictive queries Uses of WikiQuery: – Expert searchers – Classroom: becoming experts – Whenever you face a hard query 16

Source Code Based on MediaWiki version 1.17 Available at Questions: now OR catch me at lunch! 17