1. Beliefs and Biases in Web Search
Hua zhang

2. Questions:
Do people’s beliefs in different outcomes change as a result of searching?
Do search engine results bias in favor of particular outcomes?
What is the impact of bias on search outcomes, specifically on answer accuracy?

3. Outline: Brief introduction Related Work on bias and result ranking
Retrospective survey
Log analysis of search behavior
Discussion and conclusion

4. What is bias?
Bias is an inclination or outlook to present or hold a partial perspective, often accompanied by a refusal to consider the possible alternative points of view. 
Biases in cognition may lead people to create beliefs based on false premises and behave in a seemingly irrational manner.
Biases can be observed in information retrieval in situations where searchers seek or are presented with information that significantly deviates from the truth.

5. Why bias?
Cause searchers to be interested in particular results for reasons beyond relevance.
Affect aggregated behavioral signals used by search engine for ranking, result click-through
Both satisfying users’ preference and correct
Specific domain, medical

6. Why bias?
Models of search process ignore situations where seemingly irrational behavior is observed
Bias plays a central role in human judgement and decision making
Improve result relevance based on behavioral data

7. Beliefs during searching
Distribution of (a) prior and (b) posterior reported recalled beliefs about the answer (ranging from Yes to No).

8. Findings:
Leaning yes or no before searching either retain that belief, or become more certain
Unlikely to change their opinion to the opposite perspective after searching
If unsure, twice more likely to move forward to a positive answer rather than negative

9. Design search questions
Extract yes-no questions from a random sample of the logs of queries
Filter questions with medical intent from the rest
Label the questions by physicians
Base Rate: 674 questions where 55% answers are yes and 45% are no

10. Crowdsourced judgments
1. Caption judgments
Review the captions and provide one rating
Yes only
No only
Both
Neither

11. Presence and distribution
Table 4 shows a bias toward yes
Table 5 shows that results and captions containing yes are much more likely to appear than those with no

12. Crowdsourced judgments
2. Landing page judgment
Judge the full text of the results returned by the search engine
What’s next?
Analyze the SERPs generated by the search engine
Analyze the top-ranked results returned by the engine

13. Distribution of highest-ranked answers
Topmost captions and the results with yes are ranked above those with no and other answer labels.

14. Relative ordering of Yes and No
Captions with yes are ranked well above those with no for both captions and results.
Captions exhibit this bias slightly more than the results themselves
Yes was ranked above no more often than the 55% that would be expected

15. Summary
More likely to present captions/results answering a yes- no question positively (yes) in the top results
More likely to rank results with positive answers at the top of the list, and above no
distributions were skewed more positive than expected from base rates.

16. Questions?