Beliefs and Biases in Web Search Hua zhang

Questions: Do people’s beliefs in different outcomes change as a result of searching? Do search engine results bias in favor of particular outcomes? What is the impact of bias on search outcomes, specifically on answer accuracy?

Outline: Brief introduction Related Work on bias and result ranking Retrospective survey Log analysis of search behavior Discussion and conclusion

What is bias? Bias is an inclination or outlook to present or hold a partial perspective, often accompanied by a refusal to consider the possible alternative points of view.  Biases in cognition may lead people to create beliefs based on false premises and behave in a seemingly irrational manner. Biases can be observed in information retrieval in situations where searchers seek or are presented with information that significantly deviates from the truth.

Why bias? Cause searchers to be interested in particular results for reasons beyond relevance. Affect aggregated behavioral signals used by search engine for ranking, result click-through Both satisfying users’ preference and correct Specific domain, medical

Why bias? Models of search process ignore situations where seemingly irrational behavior is observed Bias plays a central role in human judgement and decision making Improve result relevance based on behavioral data

Beliefs during searching Distribution of (a) prior and (b) posterior reported recalled beliefs about the answer (ranging from Yes to No).

Findings: Leaning yes or no before searching either retain that belief, or become more certain Unlikely to change their opinion to the opposite perspective after searching If unsure, twice more likely to move forward to a positive answer rather than negative

Design search questions Extract yes-no questions from a random sample of the logs of queries Filter questions with medical intent from the rest Label the questions by physicians Base Rate: 674 questions where 55% answers are yes and 45% are no

Crowdsourced judgments 1. Caption judgments Review the captions and provide one rating Yes only No only Both Neither

Presence and distribution Table 4 shows a bias toward yes Table 5 shows that results and captions containing yes are much more likely to appear than those with no

Crowdsourced judgments 2. Landing page judgment Judge the full text of the results returned by the search engine What’s next? Analyze the SERPs generated by the search engine Analyze the top-ranked results returned by the engine

Distribution of highest-ranked answers Topmost captions and the results with yes are ranked above those with no and other answer labels.

Relative ordering of Yes and No Captions with yes are ranked well above those with no for both captions and results. Captions exhibit this bias slightly more than the results themselves Yes was ranked above no more often than the 55% that would be expected

Summary More likely to present captions/results answering a yes- no question positively (yes) in the top results More likely to rank results with positive answers at the top of the list, and above no distributions were skewed more positive than expected from base rates.

Questions?