Coverage and Independence: Measuring Quality in Web Search Results Panagiotis Takis Metaxas Lilia Ivanova Eni Mustafaraj Department of Computer Science Wellesley College, USA

Precision and Recall in Traditional IR

Precision and Recall in Web IR High Precision is easy to achieve but does not convey useful information Recall is uninteresting and cannot be computed accurately because of the enormous size of the web 85% of Web Searchers never look past top-10!

But what is Quality?

Quality when searching controversial issues?

Quality when searching Political Issues? But Google is usually so good in finding info… Why does it do that?

Define Search Quality in a web-meaningful way Comprehensive Coverage = Lack of bias towards some search results For a controversial issue (at a minimum): cover the pro, con and balanced opinions For k opinions, and top-N results: expected # of results / opinion: N/k Coverage Bias = total distance from N/k

Define Search Quality in a web-meaningful way Comprehensive Coverage = Lack of bias towards some search results (bad coverage) 0 ≤ C ≤ 1 (good coverage) Now we can talk about, e.g., 60% coverage

Define Search Quality in a web-meaningful way Independent search results = Results that are not dependent due to spamming u: URL Dependency r: Redirection Dependency c: Content Dependency l: Link Dependency

Example of Dependent Results: Google’s “HGH benefits” Redirection dependencyURL dependency Table 1: Top-10 results of Google when given the query ”HGH benefits” for August, 2007 and September, For each entry we have calculated the size of the backGraph as (|V |, |E |) revealed by the Google API and the change between these two dates.

Example of Dependent Results: Yahoo’s “Is ADHD a real disease” Link dependency Content dependency Table 4: Top-10 results of the Yahoo search engine when given the query ”Is ADHD a real disease” (August and September, 2007).

Define Search Quality in a web-meaningful way Independent search results = Results that are not dependent due to spamming u: URL Dependency r: Redirection Dependency c: Content Dependency l: Link Dependency (total dependence) 0 ≤ ≤ 1 (total independence)

Evaluating Quality of 3 Search Results Query with commercial interest: “Human Growth Hormone (HGH) benefits” Query with medical interest: “Is ADHD a real disease?” Query with political interest: “Morality of abortions”

Evaluating Quality of 3 Search Results Coverage of GoogleCoverage of YahooIndependence Our result show low coverage for controversial questions that are not highly pursued and higher coverage for an issue that is highly pursued (“Abortion”). They also show high independence of results that are not highly pursued and higher independence for an issue that is highly pursued (“Abortion”). There is significant overlap between the top-10 returns of both Yahoo and Google results!

Comparing visible neighborhoods Google Yahoo Both

Coverage and Independence: Measuring Quality in Web Search Results Panagiotis Takis Metaxas Department of Computer Science Wellesley College, USA Thank you!

Example of Dependent Results: Yahoo’s “HGH benefits”

Example of Dependent Results: Google’s “Is ADHD a real disease”

Example of Dependent Results: Google’s “Morality of Abortion”

Example of Dependent Results: Yahoo’s “Morality of Abortion”