Effects of Popularity and Quality on the Usage of Query Suggestions during Information Search Can users be induced to take bad query suggestions because they believe many others have used the suggestions in the past? Diane Kelly, Amber Cushing, Maureen Dostert, Xi Niu and Karl Gyllstrom University of North Carolina at Chapel Hill

Motivation & Background Detrimental impact of social search features Social influences on behavior – Psychology, sociology, and economics – Recommender systems, business and marketing Query suggestions as a social search feature and system usage information Users’ abilities to identify ‘good’ suggestions 2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL

Research Questions 1.Are users influenced by usage information associated with recommended queries? 2.Can users distinguish between high and low quality query suggestions? 3.What are users’ perceptions of the usefulness of query suggestions and usage information for open search tasks? 2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL

Use of query suggestions as idea tactics – A move to help users generate new approaches or solutions to information search problems – Potentially useful when user has a limited model of the problem space, is researching a difficult topic, and/or exhausts own ideas for queries Motivation & Background 2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL

Research Questions 4.What is the relationship among topic difficulty, users’ willingness to take recommendations and their abilities to distinguish between high and low quality queries? 2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL

Method Laboratory experiment with experimental search system Twenty-three undergraduate subjects Four assigned search topics (15 minutes each) Closed collection of newspaper articles (3GB, or about 1 million documents) 8 query suggestions per search topic 2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL

Method Low Quality Queries High Quality Queries High Usage Information 0-9 Low Usage Information 2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL

Outcome Measures & Analysis Suggestion Usage/Selection (Binary) Post-Search Evaluations (5-point scale, 1=low; 5=high) – Query Quality – Confidence in Rating – Willingness to Recommend to Others – Topic Difficulty Exit Interview and Manipulation Check Analysis: Logistic regression, t-tests, open-coding 2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL

Results: General Usage of Suggestions 2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL SourceNumber (%) Subject Queries425 (59%) Suggested Queries297 (41%) Total Queries722 Number of Subject Queries that Matched Query Suggestion 106

Results: Popularity Were subjects influenced by the usage information associated with recommended queries? No. Usage information (popularity) was not a significant predictor of selection behavior ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL

Results: Popularity What about subjects’ post-search evaluations of query quality? Was there a difference in ratings according to popularity? No. Subjects’ mean ratings were similar ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL PopularityMean (SD) QualityLow2.87 (1.15) High2.93 (1.18) ConfidenceLow2.90 (1.11) High2.93 (1.08) RecommendLow2.84 (1.19) High2.88 (1.23)

Results: Query Quality Could subjects distinguish between high and low quality query suggestions? Yes. Query quality was a significant predictor of selection behavior ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL

Results: Query Quality Query Quality Mean (SD) Quality*Low2.80 (1.18) High3.00 (1.15) ConfidenceLow2.84 (1.09) High2.99 (1.09) Recommend*Low2.74 (1.20) High2.89 (1.20) 2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL *p<.05 What about subjects’ post- search evaluations of query quality? Was there a difference in ratings according to actual query quality? Yes. Subjects rated high quality queries significantly higher than low quality queries.

Results: Topic Difficulty What was the relationship among topic difficulty and subjects’ willingness to take recommendations? Yes. Subjects were significantly more likely to use suggestions when the topic was hard ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL

Results: Subjects’ Perceptions What were subjects’ perceptions of the usefulness of query suggestions and usage information for open search tasks? the usage information did not influence their selections that much and was not that important for this task they ignored the usage information and did their own testing the query suggestions stimulated thinking outside the box and helped them narrow their searches the query suggestions were useful when they ran out of ideas or were unsure of how to start the search 2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL

Summary of Findings Subjects integrated the suggestions into their searching fairly quickly The usage information did not influence subjects’ selection Subjects selected more high quality queries than low quality queries and also rated these higher Subjects took significantly more suggestions when searching for difficult topics Suggestions seemed to function as an idea tactic Task type mediates the effects of social influence on selection behaviors (suggestive) 2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL

Implications Design – Tailor suggestions to properties of task (type, difficulty, and stage) Suggestion algorithms Presentation methods – Query performance prediction Method – Data collection and evaluation of suggestion use 2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL

2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL Thank You. Diane Kelly,